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Abstract 


The  INCOMMANDS  TDP  seeks  to  research,  demonstrate  and  evaluate  new  command  decision 
support  concepts  for  the  HALIFAX  Class  frigate’s  command  and  control  (C2)  system,  with  the 
objective  of  improving  team  battlespace  awareness,  and  increasing  decision  speed  and  accuracy. 
The  aim  of  this  document  is  to  support  the  design  and  development  of  Operator  Machine 
Interface  (OMI)  concepts  developed  as  part  of  the  INCOMMANDS  TDP  by  providing  a 
structured  and  comprehensive  set  of  design  guidelines  which  address  decision  aiding  concepts 
specifically.  The  guidance  within  this  document  will  be  integrated  within  a  final  document,  the 
INCOMMANDS  Human  Factors  Design  and  Evaluation  Guide,  which  will  cover  both  OMI  and 
decision  aid  guidance. 


Resume 


Le  PDT  INCOMMANDS  cherche  a  mettre  au  point,  a  et  a  evaluer  les  nouveaux  concepts  de 
soutien  a  la  decision  du  commandement  (SDC)  pour  le  systeme  de  commandement  et  de  controle 
(C2)  de  la  fregate  de  la  classe  HALIFAX  dans  le  but  de  sensibiliser  davantage  les  equipes  a  ce  qui 
se  passe  sur  le  terrain  et  de  permettre  une  prise  de  decision  plus  rapide  et  eclairee.  Le  present 
document  vise  a  appuyer  la  conception  et  l’elaboration  des  concepts  d’interface  operateur- 
machine  (IOM)  elabores  dans  le  cadre  du  PDT  INCOMMANDS  en  foumissant  un  ensemble 
structure  et  complet  de  lignes  directrices  relatives  a  la  conception  traitant  particulierement  des 
concepts  d’aide  a  la  decision.  Les  directives  contenues  dans  le  present  document  seront  integrees 
dans  un  document  final,  le  Guide  de  conception  et  devaluation  tenant  compte  des  facteurs 
humains  -  INCOMMANDS,  qui  couvrira  les  directives  relatives  a  1’IOM  et  a  l’aide  a  la  decision. 


Executive  Summary 

The  INCOMMANDS  TDP  seeks  to  research  demonstrate  and  evaluate  new  command  decision 
support  concepts  for  the  HALIFAX  Class  frigate’s  command  and  control  (C2)  system,  with  the 
objective  of  improving  team  battlespace  awareness,  and  increasing  decision  speed  and  accuracy. 

The  intention  of  this  document  is  to  support  the  design  and  development  of  Operator  Machine 
Interface  (OMI)  concepts  developed  as  part  of  the  INCOMMANDS  TDP  by  providing  a 
structured  and  comprehensive  set  of  design  guidelines  which  address  decision  aiding  concepts 
specifically.  The  guidance  within  this  document  will  be  integrated  within  a  final  document,  the 
INCOMMANDS  Human  Factors  Design  and  Evaluation  Guide,  which  will  cover  both  OMI  and 
decision  aid  guidance.  This  work  was  completed  under  contract  to  DRDC-Toronto. 

The  goal  of  the  work  produced  as  part  of  Task  6,  Human  Factors  Investigations  Support,  of 
contract  W770 1-4-3544,  is  to  inform  and  guide  the  design  and  development  of  the  OMI  concepts 
explored  and  implemented  within  the  INCOMMANDS  TDP.  In  order  to  achieve  this  goal,  the 
objectives  of  this  report  are  as  follows: 

•  To  incorporate  recommended  standards  and  guidelines  that  will  guide  and  inform  the 
design  of  OMI  and  decision  aiding  concepts  developed  within  the  INCOMMANDS 
project  so  that  they  are  consistent  with  Human  Factors  best-practice; 

•  To  provide  common  OMI  design  guidance  for  existing  decision  aids,  decision  aids  under 
development,  and  future  decision  aiding  concepts,  within  the  context  of  Maritime 
Command  and  Control  (C2),  including  Threat  Evaluation  and  Weapons  Assessment 
(TEW A);  and, 

•  To  provide  guidance,  in  terms  of  metrics  and  tools,  for  the  evaluation  of  the  OMI’s 
compliance  with  the  proposed  guidelines. 

The  structure  of  the  guidance  pertaining  to  the  design  of  decision  aids  within  this  document  will 
follow  a  Perceive-Decide-Act  classification  of  decision  aiding  (and  related)  technologies.  The 
first  section  comprises  guidance  relating  to  Electronic  Support  System  (ESS)-specific  OMI 
design  goals.  The  next  section  is  devoted  to  specific  guidance  relating  to  specific  classes  of 
decision  aids;  Information  Management  Aids  (guidelines  to  optimise  and  organise  the 
information  presented  for  efficient  information  acquisition  and  synthesis),  Decision  Making  Aids 
(guidelines  to  support  efficient  and  effective  decision  making),  and  Control  and  Action  Aids 
(guidelines  to  minimise  operator  out-of-the-loop  problems).  Finally,  the  last  two  sections  cover 
guidelines  relating  to  Design  and  Evaluation,  and  Training  and  Implementation. 

The  format  of  the  Human  Factors  guidelines  presented  within  this  document  was  designed  to 
impose  a  consistent  and  logical  format  to  assist  the  reader  in  finding  the  relevant  guideline(s) 
quickly.  Each  and  every  guideline  is  composed  of  the  following  categories: 

•  Guideline  Number.  A  unique  reference  number  given  to  each  guideline,  or  set  of 
guidelines,  to  enable  rapid  searching  for  a  particular  guideline. 

•  Guideline  Title.  A  short  title  that  summarizes  the  topic  of  the  guideline(s). 

•  Source.  A  reference  for  the  source  document(s)  from  which  the  guideline(s)  was 
(were)  taken  from. 

•  Guideline(s).  A  list  of  guidelines  relevant  to  the  topic.  These  are  worded  as  ‘shall’ 
(i.e.,  mandatory)  statements. 
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•  Discussion.  Where  relevant,  supporting  evidence  for,  and/or  further  discussion  of, 
each  guideline  is  presented  in  this  section. 

•  Evaluation  Measures  and  Methodology’.  Evaluation  measures  and  methodology  are 
provided  to  determine  compliance  with  the  guideline(s).  Evaluation  measures  relate 
to  System  Performance,  Task  Performance,  Operator  Situation  Awareness,  Operator 
Workload,  Operator  Trust,  and  so  on.  The  exact  criteria  to  which  the  components  of 
the  OM1  are  evaluated  against  (e.g.,  the  extent  to  which  the  OM1  affords  adequate 
Situation  Awareness  or  workload  ‘levels’)  should  be  determined  on  a  case-by-case 
basis;  as  such,  it  is  beyond  the  scope  this  document  to  provide  exact  values,  or 
thresholds,  to  each  and  every  evaluation  criteria  mentioned.  The  evaluation  measures 
described  in  section  3.2  of  this  report  can  be  administered  by  the  following 
methodologies:  Inspection  (e.g.,  a  visual  inspection  to  determine  that  menu  structure 
is  compliant  with  guidelines,  or  visual  measurement  of  font  size  to  determine 
compliance);  Demonstration  (e.g.,  a  walk-through  by  a  Human  Factors  professional, 
to  determine  that  uncertainty  information  regarding  system  advice  is  compliant  with 
guidelines);  and  Experimentation  (e.g.,  modeling  or  simulation  activities,  or  a 
human-in-the-loop  experimental  or  questionnaire -based  study,  led  by  a  Human 
Factors  professional,  to  determine  that  the  system  promotes  acceptable  levels  of 
operator  workload). 

•  Relationship  to  Other  Guidelines.  Relevant  guidelines  (and  the  Guideline  Number)  to 
the  topic  are  presented  here.  Cross-referencing  should  make  it  less  likely  that  related 
guidelines  are  overlooked. 

The  guidance  within  this  document  does  not  present  detailed  design  solutions  or  exact 
specifications.  As  well  as  proving  to  be  technically  difficult,  time  consuming  and  prone  to 
obsolescence  over  time,  to  do  so  would  inhibit  the  creative  scope  of  the  development  team. 
Instead,  the  document  presents  generic  guidance  to  support  the  development  of  decision  aiding 
concepts.  The  detailed  design  solutions  will  be  developed  and  evaluated  by  the  development 
teams  using,  in  part,  the  evaluation  criteria  provided  in  here.  As  successful  solutions  are 
developed,  they  should  be  captured  as  design  specifications  and  added  to  the  guidance  provided 
in  this  document.  Finally,  each  project  related  to  the  development  of  decision  aids  for  Maritime 
C2  should  refer  to  the  guidance  presented  in  this  document  or  eventually  to  the  INCOMMANDS 
Human  Factors  Design  and  Evaluation  Guide.  In  doing  so,  consistency  across  all  systems  that  are 
being  developed  should  be  achieved. 

The  vast  majority  of  evaluation  measures  described  within  this  document  require  some  form  of 
human-in-the-loop  experimentation.  This  is  due  to  a  combination  of  the  imprecision  of  the 
constructs  that  need  to  be  measured  (e.g.,  Situation  Awareness,  workload  and  trust),  the 
imprecision  of  the  measurement  tools  used  to  measure  these  constructs,  and  the  imprecision  of 
the  guidelines  themselves.  It  is  therefore  imperative  that  any  lessons  learnt  from  one  OM1 
development  project  should  be  shared  with  the  wider  community. 


Sommaire 


Le  PDT  INCOMMANDS  cherche  a  mettre  au  point,  a  experimenter  et  a  evaluer  les  nouveaux 
concepts  de  soutien  a  la  decision  du  commandement  (SDC)  pour  le  systeme  de  commandement  et 
de  controle  (C2)  de  la  fregate  de  la  classe  HALIFAX  dans  le  but  de  sensibiliser  davantage  les 
equipes  a  ce  qui  se  passe  sur  le  terrain  et  de  permettre  une  prise  de  decision  plus  rapide  et 
eclairee. 

Le  present  document  vise  a  appuyer  la  conception  et  l’elaboration  des  concepts  d’interface 
operateur-machine  (IOM)  elabores  dans  le  cadre  du  PDT  INCOMMANDS  en  foumissant  un 
ensemble  structure  et  complet  de  lignes  directrices  relatives  a  la  conception  traitant 
particulierement  des  concepts  d’aide  a  la  decision.  Les  directives  contenues  dans  le  present 
document  seront  integrees  dans  un  document  final,  le  Guide  de  conception  et  devaluation  tenant 
compte  des  facteurs  humains  -  INCOMMANDS,  qui  couvrira  les  directives  relatives  a  1’IOM  et  a 
l’aide  a  la  decision.  Ces  travaux  ont  ete  realises  en  vertu  d’un  marche  avec  RDDC  Toronto. 

L’objectif  des  travaux  produits  dans  le  cadre  de  la  tache  6,  soutien  aux  enquetes  relatives  aux 
facteurs  humains,  du  marche  W7701-4-3544  consiste  a  informer  et  a  guider  la  conception  et 
l’elaboration  des  concepts  d’lOM  analyses  et  mis  en  oeuvre  dans  le  PDT  INCOMMANDS.  A 
cette  fin,  les  objectifs  de  ce  rapport  sont  les  suivants  : 

•  Integrer  des  normes  et  des  lignes  directrices  recommandees  qui  guideront  et  fa9onneront 
la  conception  des  concepts  d’lOM  et  d’aide  a  la  decision  elabores  dans  le  cadre  du  projet 
INCOMMANDS,  de  sorte  qu’ils  correspondent  aux  meilleures  pratiques  des  facteurs 
humains; 

•  Assurer  une  direction  commune  en  matiere  de  conception  d’lOM  en  ce  qui  touche  les 
aides  a  la  decision  actuelles,  les  aides  a  la  decision  en  conception  et  les  concepts  d’aide  a 
la  decision  futurs  dans  le  contexte  de  commandement  et  controle  (C2)  de  la  Marine, 
comprenant  revaluation  de  la  menace  et  la  designation  des  armes  (TEWA); 

•  Foumir  des  directives  en  ce  qui  touche  les  mesures  et  les  outils  au  niveau  de  revaluation 
de  la  conformite  de  1’IOM  aux  lignes  directrices  proposees. 

La  structure  des  directives  relatives  a  la  conception  des  aides  a  la  decision  dans  le  present 
document  suivra  une  classification  Perceive-Decide-Act  des  technologies  d’aide  a  la  decision  (et 
technologies  connexes).  La  premiere  partie  comprend  des  directives  liees  aux  objectifs  de 
conception  d’lOM  propre  au  SSE.  La  partie  suivante  est  consacree  aux  directives  precises 
relatives  aux  categories  d’aides  a  la  decision  particulieres  suivantes  :  aides  a  la  gestion  de 
l'information  (directives  en  vue  d’optimiser  et  d’organiser  l’information  presentee  afm  que 
1’ acquisition  et  la  synthese  de  l’information  soient  efficaces),  aides  a  la  prise  de  decisions 
(directives  en  vue  d’ appuyer  une  prise  de  decision  efficace)  et  aides  au  controle  et  a  la  gestion 
(directives  en  vue  de  reduire  les  problemes  qui  depassent  l’operateur).  Enfin,  les  deux  demieres 
parties  couvrent  les  directives  relatives  a  la  conception,  a  revaluation,  a  la  formation  et  a  la  mise 
en  oeuvre. 

Le  format  des  lignes  directrices  relatives  aux  facteurs  humains  presentees  dans  ce  document  a  ete 
con 911  pour  imposer  un  format  logique  et  uniforme  de  fa9on  a  aider  le  lecteur  a  trouver 
rapidement  la(les)  ligne(s)  directrice(s)  pertinente(s).  Toutes  les  directives,  sans  exception, 
regroupent  les  categories  suivantes  : 

•  Numero  de  la  ligne  directrice.  Numero  de  reference  unique  attribue  a  chaque  ligne 
directrice,  ou  a  un  ensemble  de  lignes  directrices,  pour  effectuer  une  recherche 
rapide. 
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•  Titre  de  la  ligne  directrice.  Court  titre  resumant  le  sujet  de  la  ligne  directrice. 

•  Source.  Reference  au  document  source  a  partir  duquel  la  ligne  directrice  a  ete  tiree. 

•  Ligne(s)  directrice(s) .  Liste  des  directives  se  rapportant  au  sujet.  11  s'agit  egalement 
d'«  obligations  »  puisque  Ton  doit  y  donner  suite. 

•  Discussion.  Lorsque  pertinent,  des  preuves  ou  des  discussions  supplementaires 
concemant  chaque  directive  sont  presentees  dans  cette  partie. 

•  Methodes  et  mesures  d ’evaluation.  Les  methodes  et  mesures  devaluation  sont 
foumies  afm  d'etablir  la  conformite  avec  les  lignes  directrices.  Les  mesures 
devaluation  se  rapportentau  rendement  du  systeme,  a  l'execution  des  taches,  a  la 
connaissance  de  la  situation,  de  la  charge  de  travail  et  a  la  confiance  manifestoes  par 
I'operatcur,  et  ainsi  de  suite.  Les  criteres  exacts  d’apres  lesquels  les  composantes  de 
1’IOM  sont  evaluees  (p.  ex.,  dans  quelle  mesure  1’IOM  assure  une  bonne 
connaissance  de  la  situation  ou  des  «  niveaux  »  de  charge  de  travail  adequats)  doivent 
etre  etablis  au  cas  par  cas.  A  ce  titre,  la  portee  de  ce  document  n'est  pas  de  foumir  les 
valeurs  ou  les  seuils  exacts  pour  tous  les  criteres  d’evaluation  mentionnes.  Les 
mesures  d’evaluation  decrites  dans  la  partie  3.2  de  ce  document  peuvent  etre 
administrees  par  les  methodes  suivantes  :  Inspection  (p.  ex.,  une  inspection  visuelle 
visant  a  determiner  si  la  structure  du  menu  est  conforme  aux  lignes  directrices  ou  une 
mesure  visuelle  de  la  taille  de  la  police  visant  a  verifier  la  conformite), 

Demonstration  (p.  x.,  une  revision  structuree  effectuee  par  un  professionnel  de 
l’ergonomie  afm  de  verifier  si  les  incertitudes  quant  aux  conseils  relatifs  au  systeme 
sont  conformes  aux  lignes  directrices)  et  Experimentation  (p.  ex.,  activites  de 
modelisation  ou  de  simulation,  ou  une  etude  experimentale  tenant  compte  de 

1’ element  humain  ou  basee  sur  un  questionnaire,  effectuees  par  un  professionnel  de 
l’ergonomie  afm  de  determiner  si  le  systeme  fait  la  promotion  de  niveaux  de  charge 
de  travail  acceptables  pour  l’operateur). 

•  Relation  avec  les  autres  lignes  directrices.  Les  lignes  directrices  (et  le  numero  des 
lignes  directrices)  se  rapportant  au  sujet  sont  presentes  ici.  La  concordance  doit 
reduire  la  probabilite  d'une  omission  au  niveau  des  lignes  directrices  connexes. 

Les  directives  enoncees  dans  le  present  document  ne  presentent  pas  de  solutions  detaillees  ou  de 
specifications  exactes  concemant  la  conception.  L’elaboration  de  telles  consignes  generait  plutot 
la  liberte  de  creation  de  l’equipe  de  developpement,  en  plus  de  representer  des  difficultes  du  point 
de  vue  technique,  d’etre  couteuses  en  temps  et  de  risquer  de  devenir  desuetes  au  fil  du  temps.  Le 
document  presente  plutot  des  directives  generates  pour  appuyer  l’elaboration  des  concepts  d’aide 
a  la  decision.  Les  solutions  detaillees  quant  a  la  conception  seront  elaborees  et  evaluees  par  les 
equipes  de  developpement  a  l’aide,  entre  autres,  des  criteres  d’evaluation  enonces  dans  le  present 
document.  Des  que  des  solutions  efficaces  auront  ete  etablies,  elles  devront  etre  saisies  en  tant 
que  specifications  relatives  a  la  conception  et  ajoutees  aux  directives  foumies  dans  le  present 
document.  En  dernier  lieu,  chaque  projet  lie  a  l’elaboration  d’aides  a  la  decision  pour  le 
commandement  et  le  controle  de  la  Marine  doit  se  reporter  aux  directives  presentees  dans  le 
present  document  ou,  ulterieurement,  au  Guides  de  conception  et  devaluation  tenant  compte  des 
facteurs  humains  -  1NCOMMANDS.  Ainsi,  tous  les  systemes  elabores  devraient  etre  uniformes. 

La  vaste  majorite  des  mesures  d’evaluation  decrites  dans  le  present  document  necessitent  une 
certaine  forme  d’ experimentation  en  matiere  d’ intervention  humaine.  Cela  est  du  a  une 
combinaison  du  manque  de  precision  des  concepts  a  mesurer  (p.  ex.,  la  connaissance  de  la 
situation,  la  charge  de  travail  et  la  confiance),  du  manque  de  precision  des  outils  de  mesure 
utilises  pour  mesurer  ces  concepts  et  du  manque  de  precision  des  lignes  directrices  elles-memes. 


II  est  done  imperatif  que  toute  logon  apprise  dans  le  cadre  d’un  projet  de  developpement  d’lMO 
soit  partagee  avec  l’ensemble  de  la  collectivite. 
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1  Introduction 


The  purpose  of  the  INCOMMANDS  TDP  (Innovative  Naval  COMbat  MANagement 
Decision  Support  Technology  Demonstration  Program)  is  to  develop,  demonstrate  and 
evaluate  advanced  Above  Water  Warfare  (AWW),  Threat  Evaluation  and  Weapons 
Allocation  (TEWA)  command  decision  support  concepts  (i.e.,  decision  aids)  for  the  command 
team  of  the  Halifax  Class  Frigate  in  order  to  improve  overall  decision-making  effectiveness. 
This  work  necessitates  the  design,  development  and  evaluation  of  innovative  Operator 
Machine  Interface  (OMI)  concepts  to  support  the  operator’s  interaction  with  the  command 
decision  support  concepts  developed  by  the  project  team. 

The  intention  of  this  document  is  to  support  the  design  and  development  of  Operator  Machine 
Interface  (OMI)  concepts  developed  as  part  of  the  INCOMMANDS  TDP  by  providing  a 
structured  and  comprehensive  set  of  design  guidelines  which  address  decision  aiding  concepts 
specifically.  The  guidance  within  this  document  will  be  integrated  within  a  final  document, 
the  INCOMMANDS  Human  Factors  Design  and  Evaluation  Guide,  which  will  cover  both 
OMI  and  decision  aid  guidance.  This  work  was  completed  under  contract  to  Defence 
Research  and  Development  Canada  (DRDC)-Toronto. 

1.1  Background 

1.1.1  Overview  of  the  INCOMMANDS  TDP 

The  INCOMMANDS  TDP  seeks  to  research,  demonstrate  and  evaluate  new  command 
decision  support  concepts  for  the  HALIFAX  Class  frigate’s  command  and  control  (C2)  system 
with  the  objective  of  improving  team  battlespace  awareness,  and  increasing  decision  speed 
and  accuracy. 

Advances  in  threat  technology,  the  increasing  difficulty  and  diversity  of  air,  land,  open-ocean 
and  littoral  scenarios,  and  the  volume  of  data  and  information  to  be  processed  under  time- 
critical  conditions  pose  significant  challenges  to  tactical  C2,  in  particular  TEWA.  The 
dynamic  environment  in  which  these  activities  are  conducted  is  one  of  high  risk  and  high 
stress  as  it  includes  organised,  intelligent,  lethal  threats.  It  is  also  inherently  uncertain  due  to 
the  imprecise  and  incomplete  nature  of  sensor  data  and  intelligence,  which  places  variable  and 
unpredictable  demands  on  decision  makers.  These  and  other  factors  are  leading  to  increased 
demands  for  time -pressured  decision  making  in  highly  ambiguous  tactical  situations.  They  are 
also  contributing  to  a  rapidly  growing  data  overload  problem  for  a  ship’s  Command  Team. 
These  performance  requirements  necessitate  that  INCOMMANDS  technologies  enhance 
decision  making  under  high-workload,  high-stress  and  uncertain  conditions.  A  central  premise 
underlying  the  INCOMMANDS  TDP  is  that  Canadian  warships  will  be  required  to  have 
advanced  and  innovative  C2  decision  aid  capabilities  that  will  improve  operator  decision¬ 
making  effectiveness  in  the  context  of  TEWA.  The  intention  is  that  the  decision  aiding 
concepts  will  present  information  and  provide  decision  recommendations  in  such  a  way  as  to 
reduce  the  mental  demands  placed  on  an  operator. 

Specific  objectives  of  the  INCOMMANDS  TDP  are  to: 
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1.  Develop  and  demonstrate  advanced  Above  Water  Warfare  (AWW)  command 
decision  support  concepts  in  a  manner  that  will  assist  the  Halifax  Modernized 
Command  and  Control  System  (HMCCS)  project  define  specifications  for  TEWA 
functions  that  are  practicable  for  Canadian  industry; 

2.  Elicit  the  Canadian  Navy’s  cognitive/decision  support  and  information  requirements 
to  perform  single  ship  AWW  command  and  control; 

3.  Develop  a  flexible  and  robust  software  architecture  that  enables  the  integration  of 
heterogeneous  algorithms  and  incremental  enhancements; 

4.  Develop  a  knowledge -based  framework  that  allows  the  efficient  exploitation  of  a 
priori  information  and  improves  both  human  and  automated  TEWA  functions; 

5.  Develop  comprehensive  evaluation  methods  and  metrics  (i.e.,  measures  of 
performance  and  effectiveness)  that  permit  the  empirical  validation  and  assessment  of 
new  decision  making  aids  and  human  decision-making  effectiveness; 

6.  Develop  an  advanced  naval  C2  modeling  and  simulation  capability  that  will  be 
compatible  with,  and  of  interest  to,  the  Canadian  Forces  Maritime  Warfare  Centre 
(CFMWC);  and, 

7.  Explore  multi-ship  TEWA  concepts  in  order  to  support  the  Canadian  Navy’s 
contribution  to  the  international  Battle  Management  Command,  Control, 
Communications,  Computers  and  Intelligence  (BMC41)  project  through  a  Task  Group 
conceptual  study. 

1.1.2  INCOMMANDS  OMI  Development:  The  Role  of  Human  Factors 
Guides 

CAE  Professional  Services  (Canada)  has  been  contracted  by  Thales  Canada  to  provide  OMI 
design  concepts  for  the  decision  support  component  of  the  1NCOMMANDS  TDP  for  DRDC 
Valcartier.  This  OMI  development  work  is  guided,  in  part,  by  a  number  of  existing  Standards 
(e.g.,  M1L-STD-1472,  Section  5.14)  and  Style  Guides  that  provide  general  guidance  and 
recommendations  on  the  Took  and  feel’  of  an  interface  (e.g.,  layout,  symbology,  interaction 
methods).  For  example,  the  Command  Decision  Aiding  Technology  (COMDAT)  TDP 
developed  an  OMI  style  guide  that  provided  a  common  framework  for  new  developments  in 
command  decision  aids.  The  COMDAT  OMI  style  guide  comprises  a  discussion  of  identified 
Human  Factors  and  usability  issues  within  the  specific  command  and  control  functions  of  a 
Halifax-Class  Canadian  Patrol  Frigate  (CPF). 

1.1.3  Limitations  of  Existing  Human  Factors  Guides 

The  COMDAT  OMI  Style  Guide  and  most  its  antecedents,  while  reflecting  current  OMI 
knowledge  and  directed  at  Maritime  C2  System,  focus  primarily  on  the  Took  and  feel’  of  the 
interface  and  provide  little  guidance  on  the  use  and  design  of  decision  aids.  Since  most 
command  and  control  systems  under  development  are  expected  to  contain  considerable 
automation  and  decision  support,  it  is  important  that  existing  Human  Factors  guidance  on  the 
use  of  decision  aids  be  made  readily  available  to  OMI  designers  and  developers. 
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Another  limitation  of  style  guides  is  that  they  rarely  include  guidance  on  how  to  evaluate  an 
OM1  to  determine  its  compliance.  Often,  compliance  is  based  on  a  heuristic  evaluation  of  the 
proposed  OM1  by  a  Human  Factors  practitioner.  While  this  may  be  sufficient  for  assessing 
specific,  well-defined  aspects  of  the  display  design,  it  is  less  satisfactory  for  assessing  less 
prescriptive  and  more  abstract  guidance.  Examples  might  include  statements  such  as  “the 
system  shall  maintain  operator  Situation  Awareness”  or  “all  colours  shall  be  easily 
discriminated”.  The  addition  of  recommended  methods  for  evaluating  compliance  makes  a 
style  guide  more  useful  to  project  managers  who  call-up  a  style  guide  as  part  of  a  system 
specification. 

1.2  INCOMMANDS  Human  Factors  Design  and  Evaluation 
Guide 

1.2.1  Objectives 

The  goal  of  the  INCOMMANDS  Human  Factors  Design  and  Evaluation  Guide  is  to  inform 
and  guide  the  design  and  development  of  the  OMI  design  concepts  explored  within  the 
INCOMMANDS  TDP.  In  order  to  achieve  this  goal,  the  objectives  of  the  INCOMMANDS 
Human  Factors  Design  and  Evaluation  Guide  are  as  follows: 

•  To  incorporate  recommended  standards  and  guidelines  that  will  guide  and  inform 
the  design  of  OMI  and  decision  aiding  concepts  developed  within  the 
INCOMMANDS  project  so  that  they  are  consistent  with  Human  Factors  best- 
practice; 

•  To  provide  common  OMI  design  guidance  for  existing  decision  aids,  decision 
aids  under  development,  and  future  decision  aiding  concepts,  within  the  context 
of  Maritime  C2,  including  Threat  Evaluation  and  Weapons  Assessment  (TEW A); 
and, 

•  To  provide  general  guidance,  in  terms  of  suggested  metrics  and  tools,  for  the 
evaluation  of  a  proposed  OMI’s  compliance  with  the  guidelines  within  the  style 
guide. 

To  meet  these  objectives,  the  INCOMMANDS  Human  Factors  Design  and  Evaluation  Guide 
will  utilise  material  from  the  following  documents: 

•  The  COMDAT  OMI  Style  Guide,  which  provides  general  guidance  on  the  design 
of  the  OMI.  The  OMI  guidance  from  the  COMDAT  OMI  Style  Guide  will  be 
revised  to  reflect  the  requirements  of  the  INCOMMANDS  OMI;  and, 

•  The  current  document,  which  provides  guidance  on  the  design  of  decision  aiding 
concepts  specifically. 

1.2.2  Terminology 

Given  the  propensity  of  computer-based  systems  that  assist  the  operator  in  a  myriad  of  mental 
and  physical  activities  (e.g.,  decision  making  aids,  decision  aids,  automation,  adaptive 
automation,  intelligent  automation,  intelligent  adaptive  interfaces,  expert  systems,  knowledge- 
based  systems,  data  fusion,  and  information  fusion)  to  varying  degrees  of  autonomy  (e.g., 
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tool,  aid,  associate,  autonomous  agent)  and  sophistication  (e.g.,  assistant,  associate  or  coach), 
the  generic  term  Electronic  Support  System  (ESS)  has  been  used  in  this  report  to  refer  to  all  of 
the  systems,  and  similar,  that  are  cited  above.  The  terminology  used  in  this  document  is  as 
follows: 


Electronic  Support 
System  (ESS): 


Operator: 


A  sophisticated  computer-based  system,  which  embodies 
domain  expertise,  used  to  assist  decision  makers  in: 

•  Acquiring,  fusing,  modelling  and  displaying  information; 

•  Aiding  the  decision  maker  in  evaluating  and  integrating 
information; 

•  Presenting  a  synthesized  picture  of  the  situation; 

•  Making  decisions;  and, 

•  Implementing  a  course  of  action. 

The  human  who  uses  the  ESS  to  assist  in  the  successful 
completion  of  their  task(s). 


All  of  the  design  guidelines  within  this  document  are  written  as  recommendations  (i.e.,  shall). 
The  intent  of  the  language  is  not  to  imply  that  all  guidance  pertaining  to  the  design  of  the  ESS 
presented  in  this  document  is  required.  Rather,  the  intent  is  that  the  OM1  and  ESS  design 
follow  the  recommendations  presented  herein  where  possible.  Much  of  the  guidance  within 
this  document  was  developed  for  the  aviation  domain  and,  as  such,  its  applicability  to  the 
naval  C2  domain  must  be  carefully  considered  before  implementation.  Once  again,  the 
implementation  of  the  advice  must  be  carefully  considered  and  evaluated  using  the  evaluation 
criteria  suggested. 

1.3  Scope  of  Document 

The  guidance  within  this  document  does  not  present  detailed  design  solutions  or  exact 
specifications.  As  well  as  proving  to  be  technically  difficult,  time  consuming  and  prone  to 
obsolescence  over  time,  to  do  so  would  inhibit  the  creative  scope  of  the  development  team. 
Instead,  this  document  presents  generic  guidance  to  support  the  development  of  decision 
aiding  concepts.  The  detailed  design  solutions  will  be  developed  and  evaluated  by  the 
development  teams  using,  in  part,  the  evaluation  criteria  provided  in  this  document.  As 
successful  solutions  are  developed,  they  should  be  captured  as  design  specifications  and 
added  to  the  guidance  provided  in  this  document.  Finally,  each  project  related  to  the 
development  of  decision  aids  for  Maritime  C2  should  refer  to  the  guidance  presented  in  this 
document,  or  eventually  to  the  INCOMMANDS  Human  Factors  Design  and  Evaluation 
Guide.  In  doing  so,  consistency  across  all  systems  that  are  being  developed  should  be 
achieved. 

This  document  includes  the  following  sections: 

a.  Introduction; 

b.  Design  Guidelines  (including  guidance  for  evaluating  compliance);  and, 

c.  Conclusions  and  Next  Steps. 
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1.4  Bibliography 

The  following  references  were  used  to  formulate  the  guidance  within  this  document.  It  is 
important  to  note  that  all  the  following  documents  cite  many  other  references;  however  this 
document  does  not  cite  these  secondary  or  tertiary  references.  Instead,  the  OM1  developer  is 
encouraged  to  read  the  source  documents  if  further  clarification  or  infoimation  is  required. 


1.4.1  Military  Standards 

DEF-STD-00-25  Human  Factors  for  Designers  of  Equipment,  24  May,  1996. 

M1F-STD  1472F  Department  of  Defense  Design  Criteria  Standard  -  Human 
Engineering,  23/08/99.  (Supersedes  M1L-STD  1472  C,  D,  E). 

1.4.2  Book  Chapters 

Endsley,  M.D.  (1996).  Automation  and  Situation  Awareness.  In  R.  Parasuraman  &  M. 
Mouloua  (Eds.).  Automation  and  human  performance:  Theory  and  applications  (pp. 
163-181).Mahwah,  N.J.  Lawrence  Erlbaum. 

This  chapter  discusses  how  various  factors  -  monitoring,  passive  decision 
making,  poor  feedback  and  poor  mental  models  -  can  impact  Situation 
Awareness  and  out-of-the-loop  performance  problems  when  interacting  with 
automated  systems.  Endsley  introduces  some  guidelines  for  the  design  and 
evaluation  of  automated  systems  to  minimize  these  problems  and  thus 
allowing  the  potential  benefits  of  automation  to  be  realized. 

Zachary,  W.  W.  and  Ryder,  J.  M.  (1997).  Decision  Support  Systems:  Integrating 
decision  aiding  and  decision  training,  In  M.  G.  Helander,  T.  K.  Landauer,  and  P.  V. 
Prabhu  (Eds.).  Handbook  of  Human-Computer  Interaction,  pp  1235-1258. 

Amsterdam,  The  Netherlands:  Elsevier  Science. 

The  authors  argue  that  experts,  novices,  and  intermediate-level  individuals 
vary  not  (just)  in  the  amount  of  knowledge  but  in  the  organization  and 
representation  of  that  knowledge.  For  experts,  there  is  typically  a  high  degree 
of  commonality  imposed  by  the  domain  itself  and  by  sociology  of  knowledge 
in  the  operational  community,  whereas  novices  and  intermediate-level 
individuals  have  less  coherent  knowledge  structures  and  exhibit  more 
variability  in  knowledge  content  and  strategies.  Decision  aids  should 
therefore  be  tailored  to  the  expertise  and  skill  of  the  decision  maker.  In 
addition,  the  support  given  to  operators  of  one  level  of  expertise  should  not 
interfere  with  the  support  given  to  operators  of  other  levels  of  expertise. 

1.4.3  Technical  Reports 

Ahlstrom,  V.,  and  Longo,  K.  (2003).  Human  factors  design  standard  [HFDS  2003]. 
(Report  number  DOT/FAA/CT-03/05  HF-STD-001):  Chapter  3  —  Automation.  Atlantic 
City  International  Airport,  NJ:  DOT/FAA  Technical  Center. 

The  Human  Factors  Design  Standard  (HFDS)  provides  reference  information 
to  assist  in  the  selection,  analysis,  design,  development,  and  evaluation  of 
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new  and  modified  Federal  Aviation  Administration  (FAA)  systems  and 
equipment.  This  document  is  based  largely  on  the  1996  Fluman  Factors 
Design  Guide  (FIFDG)  produced  by  the  FAA  in  1996.  It  converts  the  original 
guidelines  document  to  a  standard  and  incorporates  updated  information, 
including  the  newly  revised  chapters  on  automation  and  human-computer 
interface.  The  updated  document  includes  extensive  reorganization  of 
material  based  on  user  feedback  on  how  the  document  has  been  used  in  the 
past.  Additional  information  has  been  also  been  added  to  help  the  users  better 
understand  tradeoffs  involved  with  specific  design  criteria.  This  standard 
covers  a  broad  range  of  human  factors  topics  that  pertain  to  automation, 
maintenance,  displays  and  printers,  controls  and  visual  indicators,  alarms, 
alerts  and  voice  output,  input  devices,  workplace  design,  system  security, 
safety,  the  environment,  and  anthropometry  documentation.  This  document 
also  includes  extensive  human-computer  interface  information. 

Aviation  Fluman-Computer  Interface  (AFIC1)  Style  Guide.  (1998).  Report  Number 
6420 1-97U/6 1223.  Prepared  by  Veridian,  Veda  Operations 

The  AFIC1  Style  Guide  presents  guidelines  to  assist  in  the  selection,  analysis, 
design,  development  and  evaluation  of  tri-service  military  aircraft  cockpits, 
with  emphasis  on  Amy  aviation.  These  guidelines  are  intended  to 
complement  and  extend  those  published  in  other  Department  of  Defense 
(DOD)  HC1  Style  Guides. 

D1SA  (1996).  Department  of  Defense  Technical  Architecture  Framework  for 
Information  Management.  Volume  8:  DoD  Human  Computer  Interface  Style  Guide 
(Version  3.0).  Washington,  DC:  Defense  Information  Systems  Agency,  Center  for 
Information  Management. 

The  Department  of  Defence  (DoD)  Fluman  Computer  Interface  Style  Guide 
provides  reference  information  to  assist  in  the  selection,  analysis,  design, 
development,  and  evaluation  of  new  and  modified  decision  aid  systems  and 
equipment.  This  guideline  covers  a  broad  range  of  human  factors  topics  that 
pertain  to  the  design  and  implementation  of  decision  aids.  This  document  is 
largely  based  on  User-Centered  Design  (UCD)  principles. 

1.4.4  Journal  Articles 

Endsley,  M.R.  and  Kaber,  D.B.  (1999).  Level  of  automation  effects  on  performance, 
situation  awareness  and  workload  in  a  dynamic  control  task.  Ergonomics,  42(3),  462- 
492. 


This  paper  details  a  study  performed  to  explore  various  levels  of  automation 
(LOA)  designating  the  degree  of  human  operator  and  computer  control  on 
overall  human/  machine  performance  within  the  context  of  a  dynamic  control 
task.  Thirty  subjects  performed  simulation  trials  involving  various  levels  of 
automation.  Several  automation  failures  occurred  and  out-of-the-loop 
performance  decrements  were  assessed.  Results  suggest  that,  in  terms  of 
performance,  human  operators  benefit  most  from  automation  of  the 
implementation  portion  of  the  task,  but  only  under  normal  operating 
conditions;  in  contrast,  removal  of  the  operator  from  task  implementation  is 
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detrimental  to  performance  recovery  if  the  automated  system  fails.  Joint 
human/system  option  generation  significantly  degraded  performance  in 
comparison  to  human  or  automated  option  generation  alone.  Lower  operator 
workload  and  higher  situation  awareness  were  observed  under  automation  of 
the  decision  making  portion  of  the  task  (i.e.,  selection  of  options),  although 
human/system  performance  was  only  slightly  improved. 

Parasuraman,  R.  and  Riley,  V.  (1997).  Humans  and  automation:  Use,  misuse,  disuse 
and  abuse.  Human  Factors,  39(2),  230-253. 

This  paper  addresses  theoretical,  empirical  and  analytical  studies  pertaining  to 
human  use,  misuse,  disuse  and  abuse  of  automation.  Understanding  of  these 
factors  can  lead  to  improved  system  design,  effective  training  methods  and 
policies  and  procedures  involving  automation  use.  Guidelines  for  the 
effective  implementation  of  automation  are  presented. 

Sheridan,  T.B.  (2000).  Function  allocation:  algorithm,  alchemy  or  apostasy? 
International  Journal  of  Human-Computer  Studies,  52,  203-216. 

The  paper  discusses  the  following  problems  of  allocation  functions  or  tasks  to 
the  human  or  the  system:  (1)  computers,  automation  and  robotics  offer  ever 
greater  capability,  but  at  the  cost  of  greater  system  complexity  and  designer 
bewilderment,  making  the  stakes  for  function  allocation  ever  higher  than 
before;  (2)  proper  function  allocation  differs  by  process  stage;  (3)  automation 
appears  most  promising  at  intermediate  complexity,  but  the  bounds  of 
“intermediate”  are  undefined;  (4)  “human-centered  design”,  while  an 
appealing  slogan,  is  fraught  with  inconsistencies  in  definition  and 
generalizability;  (5)  “naturalistic  decision-making”  and  “ecological”  design 
are  incompatible  with  normative  decision  theory;  (6)  function  allocation  is 
design,  and  therefore  extends  beyond  science;  and  (7)  living  with  the 
technological  imperative,  letting  our  evolving  machines  show  us  what  they 
can  do,  acceding  or  resisting  as  the  evidence  becomes  clear,  appears 
inevitable. 

1.4.5  Conference  Proceedings 

Hutchins,  S.G.,  Morrison,  J.G.,  and  Kelly,  R.T.  (1999).  Principles  for  aiding  complex 
military  decision  making.  In  Proceedings  of  the  Second  International  Command  and 
Control  Research  and  Technology >  Symposium.  Monterey,  CA. 

Details  the  design  and  testing  of  a  decision  support  system  that  was 
developed  as  part  of  the  Tactical  Decision  Making  Under  Stress  (TADMUS) 
program.  The  central  hypothesis  for  the  research  is  that  presenting  decision 
makers  with  decision  support  tools  which  were  designed  to  parallel  the 
cognitive  strategies  employed  by  experts,  as  observed  in  naturalistic  settings, 
will  reduce  the  number  of  decision  making  errors.  Topics  include  a  discussion 
of:  (1)  the  theoretical  background  for  the  TADMUS  program;  (2)  a 
description  of  the  cognitive  tasks  performed;  (3)  the  decision  support  and 
human-  system  interaction  design  principles  incoiporated  to  reduce  the 
cognitive  processing  load  on  the  decision  maker;  and  (4)  a  brief  description  of 
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the  types  of  errors  made  by  decision  makers  and  interpretations  of  the  cause 
of  these  errors  based  on  the  cognitive  psychology  literature. 


1.5  Relationship  to  Other  Documents 

The  following  documents  are  directly  relevant  to  this  report. 

1.5.1  INCOMMANDS  Human  Factors  Engineering  Program  Plan  (HFEPP) 

The  1NCOMMANDS  Human  Factors  Engineering  Program  Plan  (HFEPP)  (Baker, 
Banbury  and  McIntyre,  2006)  outlines  the  approach  for  the  Human  Factors  aspects  of 
the  1NCOMMANDS  project  (i.e.,  analysis,  OM1  design,  and  verification). 

1.5.2  INCOMMANDS  Mission,  Function  and  Task  Analysis  of  TEWA 
Operations  in  Halifax-class  Ships 

The  1NCOMMANDS  MFTA  document  (Baker,  Banbury  and  McIntyre,  2006) 
describes  the  methodology  and  the  results  of  the  Mission,  Function  and  Task  Analysis 
(MFTA)  that  was  conducted  on  the  integration  of  the  INCOMMANDS  TEWA 
system  into  the  HALIFAX  Class  ships.  The  intent  of  this  report  is  to  focus  on  the 
primary  users  Operations  Room  Officer  (ORO)  and  Sensor  Weapons  Controller 
(SWC)  of  the  INCOMMANDS  system. 

1.5.3  INCOMMANDS  Operator  Machine  Interface  Concepts 

The  INCOMMANDS  Operator  Machine  Interface  (OM1)  document  (Baker,  Banbury 
and  McIntyre,  2006)  presents  a  concept  of  operations  (CONOPS)  as  well  as 
preliminary  design  concepts,  and  the  associated  rationale,  for  the  INCOMMANDS 
decision  support  OM1.  The  intent  is  to  document  the  OM1  design  sufficiently  for 
Thales  to  be  able  to  start  the  prototyping  process,  and  to  document  traceability  from 
the  analysis  results  through  to  the  OM1  design. 

1.5.4  INCOMMANDS  Demonstration  and  Experimentation  Plan  (DEP) 

The  INCOMMANDS  DEP  outlines  the  experimentation  objectives  and  structured 
methodology  that  will  be  used  to  assess  and  demonstrate  the  impact  of  a  prototype 
Command  Decision  Support  Concept  (CDSC)  capability  on  the  performance  of 
Threat  Evaluation  and  Weapons  Assignment  (TEWA)  activities  conducted  on  the 
Halifax  Class  Frigate. 

1.5.5  Command  Decision  Aiding  Technology  (COMDAT)  OMI  Style 
Guide 

The  COMDAT  OMI  Style  Guide  Version  1.0,  published  in  2001,  defines  the  overall 
Took  and  feel’  philosophy  of  the  OMI  screens  and  functions  to  support  development 
of  Command  Decision  Aids  for  Halifax-Class  Canadian  Patrol  Frigates  (CPFs).  The 
COMDAT  OMI  Style  Guide  will  be  combined  with  the  guidance  pertaining  to  the 
design  of  decision  aiding  concepts  within  this  document  to  create  the 
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INCOMMANDS  Human  Factors  Design  and  Evaluation  Guide.  The  OMI  guidance 
from  the  COMDAT  OMI  Style  Guide  will  be  revised  to  reflect  the  requirements  of 
the  INCOMMANDS  OMI. 
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2  Electronic  Support  Systems 


The  following  section  describes  the  rationale  for  the  structure  of  the  guidance  pertaining  to  the 
design  of  ESSs  that  support  decision  making.  The  large  number  and  diversity  of  the 
guidelines  relevant  to  the  design  of  OMls  for  ESSs  presents  a  significant  challenge  to  the 
development  of  an  appropriate  structure  that  facilitates  rapid  access  by  the  reader  to  the 
relevant  guidelines. 

2.1  Definition 

Electronic  support  systems  are  systems  that  provide  support  to  human  decision-making 
processes,  either  unsolicited  or  by  operator  request.  Electronic  Support  Systems  can  narrow 
the  decision  alternatives  down  to  a  few  plausible  options  or  suggest  a  preferred  decision  based 
on  the  available  data.  Advanced  computer-based  ESSs  and  displays  represent  a  promising 
means  of  supporting  decision  makers  in  complex,  dynamic  and  uncertain  environments.  For 
example,  they  can  help  alleviate  the  high  levels  of  cognitive  and  collaborative  demands  placed 
on  operators  working  within  these  types  of  environment.  Such  technological  solutions  should 
also  be  instrumental  in  helping  operators,  both  individually  and  as  a  team,  respond  in  an 
increasingly  agile,  adaptive  and  effective  manner  in  the  face  of  growing  complexity,  novelty 
and  change. 

2.2  Limitations 

Complex  information  gathering  and  processing  systems  have  been  designed  to  aid  the 
decision-maker  in  the  past.  Elowever,  these  systems  often  increase  the  decision-maker’s 
burden  due  to  the  inherent  system  complexity  and  the  failure  to  design  them  in  a  way  that 
mitigate  the  cognitive  processing  limitations  of  the  operator,  or  in  a  way  that  ensures 
compatibility  between  the  system  and  the  tasks  conducted  by  the  operator.  Often,  these 
systems  require  operators  to  perform  difficult  cognitive  tasks  under  intense  levels  of 
workload.  They  must  perceive,  synthesize  and  determine  the  relevance  of  a  continual  stream 
of  incoming  information,  often  pertaining  to  several  concurrent  entities,  while  projecting 
future  anticipated  events  and  making  decisions  regarding  actions  to  be  taken.  The  result  is  that 
the  decision  makers  may  fail  to  remember  or  overlook  critical  pieces  of  information,  lose 
awareness  of  the  situation  and  make  hurried  decisions  that  produce  flawed  and  ineffective 
courses  of  action.  An  effective  ESS,  therefore,  should  synthesize  much  of  the  information 
used  to  develop  situation  awareness  and  present  a  coherent  picture  of  the  situation  to  the 
operator  by  performing  many  of  the  cognitive  processing  tasks  on  behalf  of  the  operator  and 
presenting  fused  and  integrated  information  to  the  operator  (Flutchins,  Morrison  and  Kelly, 
1999).The  design  of  an  ESS  OM1,  and  its  related  functionality,  is  therefore  critical  to  support 
operator  decision  making.  A  set  of  guidelines  for  use  as  a  reference  during  OM1  design 
development  should  ensure  that  the  system  is  able  to  support  the  decision  making  processes  of 
the  operator,  as  well  as  promoting  high  levels  of  operator  trust  and  acceptance  of  the  system. 
The  guidance  within  this  document  therefore  comprises  guidelines  relevant  to  aiding  and 
supporting  operator  decision  making  in  uncertain,  novel,  or  time-critical  environments. 
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2.3  Taxonomic  Classification  of  ESS  Technologies 

In  order  to  structure  the  guidelines  within  this  document  a  conceptual  framework  based  on  the 
work  of  Neisser  was  used.  Neisser  (1976)  formulated  the  concept  of  a  ‘Perceptual  Cycle’, 
whereby  the  interaction  between  the  human  and  his  or  her  environment  shapes  the  human’s 
perceptions,  decisions  and  actions  (see  Figure  1).  In  this  view,  cognition  is  a  continuous  cycle 
of  perception,  decision  and  action  whereby  these  processes  occur  in  parallel  and  with  different 
foci.  Each  of  these  processes  provides  both  cognitive  limitations  and  unique  human  strengths. 
Similar  frameworks  can  be  found  in  the  Observe  Orient  Decide  Act  (OODA)  loop  (Boyd, 
1976),  the  Perceptual  Control  Theory  (Powers,  1973)  and  models  of  Situation  Awareness  in 
dynamic  decision  making  (Endsley,  1996). 


Figure  1:  The  Perceptual  Cycle  (Neisser,  1976). 

The  focus  of  the  guidance  within  this  document  is  to  provide  support  to  the  development  of 
tools  to  aid  the  operator  make  decisions,  as  well  as  to  provide  support  to  the  development  of 
tools  that  aid  the  operator  acquire  and  manage  information  and  instigate  a  course  of  action. 

For  example,  an  effective  decision  aid  should  synthesize  and  present  relevant  information  in 
such  a  way  as  to  provide  the  operator  with  an  accurate  and  coherent  picture  of  the  situation  so 
that  effective  and  timely  decision  can  be  made.  In  addition,  the  system  should  also  assist  the 
operator  in  performing  the  determined  course  of  action. 

The  breadth  of  how  computer-based  systems  can  assist  the  operator  in  a  myriad  of  mental  and 
physical  activities  is  reflected  by  the  large  variety  of  names  given  to  these  systems.  As 
discussed  in  section  1.2.2  of  this  report,  the  generic  term  Electronic  Support  System  (ESS)  has 
been  used  in  this  report  to  refer  to  all  of  the  systems,  and  similar,  that  are  cited  above. 
Furthermore,  these  systems  can  be  classified  within  the  Neisser’s  perceptual  cycle  (see  Table 
1). 

In  line  with  the  Perceive-Decide-Act  structure,  ESSs  can  be  classified  in  terms  of: 
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2.3.1  Information  Management  Aids  (‘Perceive’) 

Systems  include  information  acquisition  and  integration  (e.g.,  data  and  information 
fusion).  This  type  of  aiding  includes  fdtering,  distributing  or  transforming  data, 
providing  confidence  estimates  and  integrity  checks,  and  enabling  Operator  requests. 
Finally,  Information  Acquisition  Aids  might  also  manage  how  this  information  is 
presented  to  the  Operator. 

2.3.2  Decision  Making  Aids  (‘Decide’) 

These  systems  provide  support  to  human  decision  making  processes,  either 
unsolicited  or  by  operator  request,  by  narrowing  the  decision  alternatives  down  to  a 
few  plausible  options,  or  by  suggesting  a  preferred  decision  based  on  the  available 
data. 

2.3.3  Control  and  Action  Aids  (‘Act’) 

These  systems  execute  actions  or  control  tasks  with  some  degree  of  autonomy. 


Table  1:  The  taxonomic  classification  of  Electronic  Support  Systems  (ESSs)  in  terms  of 
their  support  to  operator  perception,  decision  making  and/or  action. 


PERCEIVE 

DECIDE 

ACT 

Data  Fusion 

Decision  Support 
System  (DSS) 

Conventional 
Automation  (e.g., 
autopilot) 

Information  Fusion 

Decision  Aid 

Adaptive 

Automation 

Intelligent  Adaptive 
Interfaces 

Knowledge -Based 
System  (KBS) 

Expert  System 

Intelligent  Aiding 

Associate  Systems  Technology 

12 


3  Decision  Aid  Guidance 


3.1  Guidance  Structure 

The  structure  of  the  guidance  within  this  document  follows  the  Perceive-Decide-Act 
classification  of  ESS  technologies.  The  first  section  comprises  guidance  relating  to  ESS- 
specific  OM1  design  guidelines  (more  general  OM1  guidance  can  be  found  in  the  COMDAT 
OM1  Style  Guide  and  in  Annex  A  of  this  report).  The  next  section  is  devoted  to  specific 
guidance  relating  to  specific  classes  of  decision  aids;  Information  Management  Aids 
(guidelines  to  optimise  and  organise  the  information  presented  for  efficient  information 
acquisition  and  synthesis),  Decision  Making  Aids  (guidelines  to  support  efficient  and 
effective  decision  making),  and  Control  and  Action  Aids  (guidelines  to  minimise  operator 
out-of-the-loop  problems).  Finally,  the  last  two  sections  of  the  Style  Guide  cover  guidelines 
relating  to  Design  and  Evaluation,  and  Training  and  Implementation. 

The  structure  is  summarized  below: 

3.3  ESS  OM1  Design  Guidelines 

3.3.1  General  Design  Goals 

3.3. 1 . 1  Employ  Operator-Centered  Principles 

3. 3. 1.2  Optimize  Eluman-System  Interaction 

3. 3. 1.3  Promote  ESS  Robustness  and  Resilience  to  Operator  Error 

3.3. 1 .4  Support  Operator  Monitoring  of  ESS  Functioning 

3.3.2  Employ  Operator-Centered  OM1  Design 

3.3.3  Support  Different  Modes  of  Operation 

3.3.4  Provide  System  Response  and  Feedback 

3.3.5  Support  Identification  and  Management  of  ESS  Faults  and  Failures 

3.4  Class-specific  Guidelines 

3.4.1  Information  Management  Aids 

3 .4. 1 . 1  Optimize  Information  Presentation  and  Information  Management 

3.4. 1 .2  Optimize  Display  of  Information 

3.4.2  Decision  Making  Aids 

3.4.2. 1  Ensure  Appropriate  Implementation 

3. 4.2. 2  Support  Decision  Making  Strategies 

3. 4.2. 3  Keep  Operators  in  Control 

3. 4.2. 4  Maximize  Operator  Situation  Awareness  by  Increasing  System 
Transparency 

3.4.3  Control  and  Action  Aids 
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3.4.3. 1  Keep  Operators  ‘In-the-Loop’ 

3.5  Design  and  Evaluation 

3.5. 1  Adopt  Operator-Centered  Design  Principles 

3.5.2  Adopt  Operator-Centered  Evaluation  Principles 

3.6  Training  and  Implementation 

3.6. 1  Manage  Introduction  of  the  ESS 

3.6.2  Train  to  Overcome  ‘Automation  Biases’ 

3.6.3  Train  to  Overcome  ESS  Failures 

3.6.4  Promote  Operator  Acceptance  and  Trust  in  ESSs 

3.1.1  Example  of  Guideline  Format 

The  following  section  describes  the  format  of  each  and  every  Human  Factors  guideline 
presented  within  the  Style  Guide.  The  intention  was  to  impose  a  consistent  and  logical  format 
on  each  guideline  to  assist  the  reader  in  finding  the  relevant  guideline(s)  quickly.  The  format 
is  illustrated  in  Table  2,  and  the  legend  is  described  below: 


Table  2:  Example  of  Human  Factors  guidelines  within  the  Style  Guide. 


1.1  © 

Provide  accurate  and  reliable  information  © 

Source:  HFDS,  2003  /  Martel,  1996  © 

Guideline(s): 

© 

1. 

The  ESS  shall  provide  accurate  and  reliable  information.  That  is,  the  correctness  of  the 
information  in  reflecting  the  reality. 

Discussion: 

© 

1.  Accurate  and  reliable  information  contributes  to  the  Operator’s  trust  in  the  ESS,  supports 
the  decision  making  process  and  increases  the  probability  of  an  appropriate  course  of 
action  being  chosen. 


Evaluation  Measures  and  Methodoloqy:  (6) 

Inspect 

Demonstrate 

Experiment 

System  Performance: 

•  Percentage  of  tracks  correctly  identified 

□ 

0 

□ 

□ 

0 

□ 

•  Percentage  of  tracks  correctly  identified 

□ 

0 

□ 

•  Percentage  of  false  alarms  and  misses 

□ 

0 

□ 

•  Age  of  information 

Relationship  to  Other  Guidelines:  @ 

•  If  accurate  and  reliable  information  can  not  be  presented  to  the  Operator,  refer  to 
Guideline  1 .2.3 
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©  Guideline  Number.  A  unique  reference  number  given  to  each  guideline,  or  set  of 
guidelines,  to  enable  rapid  searching  for  a  particular  guideline. 

©  Guideline  Title.  A  short  title  that  summarizes  the  topic  of  the  guideline(s). 

©  Source.  A  reference  for  the  source  document(s)  from  which  the  guideline(s)  was 
(were)  taken  from. 

©  Guideline(s) .  A  list  of  guidelines  relevant  to  the  topic.  These  are  worded  as  ‘shall’ 
(i.e.,  mandatory)  statements. 

©  Discussion.  Where  relevant,  supporting  evidence  for,  and/or  further  discussion  of, 
each  guideline  is  presented  in  this  section. 

©  Evaluation  Measures  and  Methodology.  Evaluation  measures  and  methodology  are 
provided  to  determine  compliance  with  the  guideline(s).  Evaluation  measures  relate  to 
System  Performance,  Task  Performance,  Operator  Decision  Quality,  Operator 
Situation  Awareness,  Operator  Workload,  Operator  Acceptance  and  Trust,  and 
Operator  Perceptions  ofUsabilty  and  Utility.  These  measures  are  described  in  section 
3.2.  The  exact  criteria  to  which  the  components  of  the  OMI  are  evaluated  against 
(e.g.,  the  extent  to  which  the  OMI  affords  adequate  Situation  Awareness  or  workload 
‘levels’)  should  be  determined  on  a  case-by-case  basis;  as  such,  it  is  beyond  the  scope 
this  document  to  provide  exact  values,  or  thresholds,  to  each  and  every  evaluation 
criteria  mentioned.  The  evaluation  measures  described  in  the  next  section  can  be 
administered  by  the  following  methodologies:  Inspection  (e.g.,  a  visual  inspection  to 
determine  that  menu  structure  is  compliant  with  guidelines,  or  visual  measurement  of 
font  size  to  determine  compliance);  Demonstration  (e.g.,  a  walk-through  by  a  Human 
Factors  professional,  to  determine  that  uncertainty  information  regarding  system 
advice  is  compliant  with  guidelines);  and  Experimentation  (e.g.,  modeling  or 
simulation  activities,  or  a  human-in-the-loop  experimental  or  questionnaire -based 
study  to  determine  that  the  system  promotes  acceptable  levels  of  operator  workload). 

©  Relationship  to  Other  Guidelines.  Relevant  guidelines  (and  the  Guideline  Number) 
to  the  topic  are  presented  here.  Cross-referencing  should  make  it  less  likely  that 
related  guidelines  are  overlooked. 


3.2  Evaluation  Measures 

The  following  system-based  and  operator-based  measures  of  effectiveness  were  derived  from, 
in  part,  the  INCOMMANDS  Demonstration  and  Experimental  Plan  (DEP). 

3.2.1  System-based  Measures  of  Effectiveness 

The  following  measures  of  system-level  effectiveness  are  illustrative  of  system-level  (i.e., 
human-machine)  measures  of  effectiveness: 

•  Time  to  detect  targets,  taken  from  the  time  the  target  was  available  for  detection  by 
the  system’s  sensors  to  the  time  of  actual  threat  determination; 
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•  Percentage  of  targets  detected,  through  a  comparison  of  system-determined  targets 
versus  ground  truth  in  any  experimental  situation; 

•  Percentage  of  targets  recognized/identified,  through  a  comparison  of  the  identification 
of  detected  targets  versus  ground  truth  in  any  experimental  situation;  and, 

•  Accuracy  of  target  location  of  all  target  detection/recognition/identification,  through  a 
comparison  of  reported  positions  versus  ground  truth. 

3.2.2  Operator-based  Measures  of  Effectiveness 

The  following  measures  of  operator  performance  were  derived  form  the  1NCOMMANDS 
Mission,  Function  and  Task  Analysis  of  TEW  A  Operations  in  Halifax-class  Ships  and  are 
described  in  more  detail  within  the  1NCOMMANDS  Demonstration  and  Experimentation 
Plan  (DEP). 

3. 2. 2. 1  Operator  Performance 

3.2.2. 1 .1  Speed  of  Task  Completion 

Measurement  of  Task  Performance  Speed  can  be  used  to  provide  an  objective  measure  of 
operator  performance  for  pre-defmed  sequences  of  events  (e.g.,  time  taken  to  detect  target, 
time  taken  to  identify  target,  time  taken  to  apply  combat  power).  Determination  of  task 
performance  speed  can  be  measured  manually  by  means  of  a  stopwatch  during  real  time  or 
later  by  means  of  video  analysis.  Task  performance  speed  can  also  be  determined 
automatically  by  means  of  time  a  marker  in  the  prototype  evaluation  software  which 
determines  the  time  increment  between  predefined  task  start  and  stop  times. 

3.2.2. 1 .2  Accuracy  of  Task  Performance 

Measurement  of  Task  Performance  Accuracy  can  be  used  to  provide  an  objective  measure  of 
operator  performance  of  pre-defmed  sequences  of  events  (e.g.,  correct  target  identification, 
correct  application  of  combat  power).  Determination  of  task  performance  accuracy  can  be 
measured  by  comparing  test  participant  performance  and  output  against  pre-determined 
performance  criteria. 

3. 2.2.2  Operator  Decision  Making  Quality 

One  difficulty  with  conducting  research  on  operator  decision  making  is  deciding  how 
‘performance’  is  defined.  For  example,  the  notion  of  what  constitutes  a  ‘correct’  decision  is 
highly  subjective  and  context-dependent.  Research  to-date  has  been  dominated  by  assessing 
the  quality  of  a  decision  by  quantifying  the  value  of  the  outcome  for  a  given  event.  In  many 
instances  this  will  be  a  binary  output  in  terms  of  a  simple  ‘correct’  or  ‘incorrect’  outcome. 
Unfortunately,  such  a  measure  reveals  little  of  the  decision  processes  involved  in  arriving  at  a 
course  of  action;  indeed,  a  correct  decision  could  have  been  reached  by  guess-work  alone  (see 
Clark,  Banbury,  Richards  and  Dickson,  2004,  for  a  discussion). 

3. 2. 2. 2.1  Situation  Assessment:  Decision  ‘Substrate’ 

It  is  important  to  consider  decision  making  in  the  context  of  ‘process’  (i.e.,  how  a  decision 
was  reached),  rather  than  ‘outcome’  (i.e.,  what  decision  was  reached).  Inherent  in  the 
‘recognition-primed’  accounts  of  decision  making  is  the  notion  of  pattern-matching  the 
mental  representation  of  the  situation  with  past  experience.  Clearly,  the  quality  of  decision 


16 


making  is  directly  related  to  the  quality  of  this  mental  representation  of  the  situation,  which  in 
turn  is  directly  related  to  the  quality  of  the  processes  undertaken  to  acquire  it.  Thus,  one  way 
of  assessing  decision  quality  is  to  assess  the  quality  of  the  situation  assessment  processes 
underlying  the  formation  of  the  decision  (i.e.,  the  decision  ‘substrate’;  see  Clark,  Banbury, 
Richards  and  Dickson,  2004). 

For  example,  Banbury,  Dudfield,  Floermann  and  Soli,  (in  press)  developed  a  questionnaire- 
based  measure  to  assess  the  efficacy  of  the  situation  assessment  processes.  The  Factors 
Affecting  Situation  Awareness  (FASA)  questionnaire  comprises  30  questions,  divided  into  the 
following  five  sub-scales:  Attention  Management  (participants’  ability  to  attend  to  more  than 
one  task  at  a  time  and  resume  a  task  successfully  after  being  interrupted);  Information 
Management  (participants’  motivation  to  acquire  appropriate  information  to  make  rational 
decisions);  Cognitive  Efficiency  (participants’  ability  to  ignore  distractions  and  maintain  SA 
despite  external  stressors);  Automaticity  (participants’  experience  of  performing  routine  tasks 
in  a  highly  practiced,  automatic  way),  and  Inter-Personal  Dynamics  (participants’  knowledge 
of  non-verbal  communication  and  their  views  on  what  team  membership  entails).  The  FASA 
questionnaire  was  used  in  the  assessment  of  a  bespoke  SA  training  program  for  commercial 
airline  pilots  by  providing  a  more  diagnostic  measure  of  aircrews’  acquisition  and 
maintenance  of  SA. 

3. 2. 2. 2. 2  Timeliness  and  Agility 

With  the  nature  of  modem  warfare  relying  heavily  on  information  quality  and  supremacy,  it 
has  become  crucial  that  operators  are  able  to  quickly  adapt  to  the  changing  context  within 
which  they  may  find  themselves.  The  speed  of  decision  making  can  be  couched  in  terms  of 
decision  ‘timeliness’  (e.g.,  a  change  of  strategy  at  the  appropriate  time)  and  decision  ‘agility’ 
(e.g.,  the  ability  to  be  adaptable  to  changing  circumstances). 

3. 2. 2. 2. 3  Consistency 

The  consistency  of  decision  making  is  a  useful  insight  into  decision  making  quality,  as  it  can 
be  argued  that  differences  of  outcome  from  decisions  based  on  the  same  data  could  indicate 
inappropriate  or  incorrect  reasoning  processes.  Arguably,  operators  who  have  made  accurate 
situation  assessments  and  correct  inferences  based  on  these  data,  should  reach  the  same 
outcome  each  and  every  time  a  similar  decision  is  made. 

Banbury,  Selcon,  Endsley,  Gorton  and  Tatlock  (1998),  investigated  how  pilot  decision 
making  is  affected  by  the  manner  in  which  Combat  identification  decision  aid  information 
regarding  system  reliability  is  presented,  by  requiring  aircrew  participants  to  respond  to  a 
machine-identified  target  with  a  “shoot/no  shoot”  decision.  The  study  investigated  whether 
the  provision  of  an  alternate  option  to  the  primary  identification  would  affect  the  decision  to 
shoot,  especially  if  this  secondary  option  was  either  another  enemy  aircraft  or  a  friendly 
fighter.  In  addition,  two  different  representations  were  evaluated;  one  in  which  the 
information  was  presented  as  system  uncertainty;  and  the  other  in  which  it  was  presented  as 
system  confidence.  The  results  indicated  that  decision  making  behaviour  changed  when  the 
system  explicitly  identified  a  friendly  aircraft  as  the  secondary  target  -  prior  willingness  to 
fire  on  a  target  with  a  relatively  high  level  of  uncertainty  disappeared.  The  time  taken  to  make 
the  decision  was  also  found  to  be  mediated  by  what  information  was  given.  The  use  of 
consistency  as  a  measure  of  decision  quality  therefore  has  some  merit;  participants  were  not 
able  to  reach  the  same  decision  to  shoot,  even  though  they  were  given  the  same  information, 
albeit  presented  in  different  ways. 
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3. 2. 2. 2. 4  Justifiability  and  Rationality 

Both  justifiability  and  rationality  serve  to  compliment  the  accuracy  measure  of  decision 
making  quality  in  that  they  provide  further  insight  into  the  underlying  processes  that  led  the 
operator  to  make  a  decision.  Specifically,  operators  that  make  an  “inaccurate”  decision  may 
be  able  to  fully  justify  their  choice  by  providing  the  rationale  behind  their  decision.  The 
quality  of  this  decision  is  therefore  good,  despite  its  seemingly  inaccurate  nature.  Further,  an 
“accurate”  decision  is  not  necessarily  a  good  one  given  that  it  could  have  been  made  by 
chance  alone  (i.e.,  the  operator  guessed  right).  If  the  operator  was  asked  why  they  made  this 
decision,  they  would  not  be  able  to  justify  it  or  provide  a  corresponding  rationale.  In  this  case 
then,  accuracy  alone  would  provide  an  incomplete  and  misleading  metric  of  the  quality  of  the 
decision  that  was  made. 

3. 2.2.3  Operator  Situation  Awareness 

Situation  Awareness  (SA)  is  a  key  determinant  of  task  performance  and  relates  to  the  ability 
of  the  operator  to  maintain  awareness  of  task-relevant  objects  in  their  immediate  environment. 
The  measurement  of  SA  within  the  context  of  the  1NCOMMANDS  TD  project  is  important 
given  the  importance  placed  on  automation-based  technologies.  On  one  hand,  these 
technologies  might  afford  the  operator  more  information  than  they  might  otherwise  have 
access  to  resulting  in  higher  levels  of  operator  SA.  On  the  other  hand,  it  is  possible  that  too 
much  reliance  on  these  automation  technologies  might  have  a  negative  impact  of  operator’s 
SA  because  the  operator  is  consigned  to  monitoring  the  automation,  and  not  fully  engaged  in 
the  task. 

3. 2. 2. 3.1  Situation  Awareness  Global  Assessment  Technique  (SAGAT) 

SAGT  is  an  objective  measure  of  Situation  Awareness.  SAGAT  employs  periodic,  randomly 
timed  freezes  in  a  simulation  scenario  during  which  all  of  the  operators  displays  are 
temporarily  blanked.  At  the  time  of  the  freeze  a  series  of  queries  are  provide  to  the  operator  to 
assess  his  or  her  knowledge  of  what  was  happening  at  the  time  of  the  freeze.  The  queries 
typically  cover  SA  elements  on  all  three  levels  of  SA: 

•  Perception  (i.e.,  noticing  all  of  the  relevant  entities  in  the  environment); 

•  Comprehension  (i.e.,  understanding  their  meaning);  and, 

•  Projection  (i.e.,  anticipating  their  future  states). 

Queries  are  determined  based  on  an  in-depth  cognitive  tasks  analysis  that  must  be  conducted 
for  each  domain  SAGAT  is  used  in.  The  questions  are  typically  a  random  subset  of  a  larger 
standard  set  of  questions  that  are  relevant  to  the  training  scenario.  Operator’s  responses  to 
these  queries  are  scored  based  on  what  was  actually  happening  in  the  simulation  at  the  time  of 
each  freeze. 

The  main  advantage  of  SAGAT  is  that  it  allows  an  objective  unbiased  index  of  SA  that 
assesses  operator  SA  across  a  wide  range  of  elements  that  are  important  for  SA  in  a  particular 
system.  The  main  disadvantage  of  SAGAT  is  that  it  requires  freezes  in  the  simulation,  and  as 
a  result  this  measure  can  only  be  used  for  the  laboratory  evaluations.  Because  the  freezes  are 
random  and  over  such  a  broad  spectrum  of  operator  SA  requirements,  operators  cannot 
prepare  for  the  queries  and  it  has  been  found  that  the  freezes  do  not  affect  performance  in  the 
simulations. 
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An  alternative  SAGAT  approach  is  to  measure  the  amount  of  time  it  takes  a  subject  to  report 
anomalies  embedded  into  the  scenario,  then  note  the  time  it  takes  to  deviate  from  the  original 
plan,  given  the  change  in  circumstances. 

SAGAT  can  provide  unique  insight  into  crew  performance  within  simulated  team 
environments  as  well  as  individual  performance.  Queries  can  be  designed  to  assess  specific 
SA  requirements  for  each  team  member  role.  More  importantly,  however,  responses  to 
queries  related  to  common  SA  requirements  can  be  compared  across  team  members, 
identifying  SA  differences  between  team  member  roles.  In  addition,  specific  responses  can  be 
compared  to  determine  whether  the  same  responses  (correct  or  incorrect)  are  made  across 
team  member  roles.  This  type  of  analysis  can  provide  diagnostic  information  regarding  the 
source  of  breakdowns  in  team  SA.  For  example,  common  incorrect  responses  may  be 
indicative  of  problems  that  affect  the  entire  team  in  a  similar  way  (such  as  poorly  designed 
information  display).  Alternatively,  a  mix  of  correct  and  incorrect  responses  or  different 
incorrect  responses  across  team  member  roles  may  be  indicative  of  breakdowns  in  team 
coordination. 

3. 2. 2. 3. 2  Situation  Awareness  Rating  Technique  (SART) 

The  Situation  Awareness  Rating  Technique  (SART)  provides  a  validated  and  practical 
subjective  rating  tool  for  the  measurement  of  SA,  based  on  personal  construct  dimensions 
associated  with  SA.  The  structure  of  the  construct  dimensions  has  been  interpreted  as 
comprising  three  related  conceptual  groups,  which  form  the  principal  dimensions  of  SART, 
namely: 


•  Demand  for  Attentional  Resources  or  D  (complexity,  variability,  instability); 

•  Supply  of  Attentional  Resources  or  S  (arousal,  concentration,  division  of  attention, 
spare  mental  capacity); 

•  Understanding  of  the  situation  or  U  (information  quality,  information  quantity, 
familiarity). 

The  most  commonly-used  version  of  SART  is  the  14-dimension  version  (see  Figure  2). 
Instead  of  numeric  Likert-scales,  a  graphical  display  of  the  rating  scales  is  utilized,  where  the 
length  of  line  from  the  left  hand  side  of  the  scale  to  the  participant’s  mark  (in  millimetres) 
represents  a  respective  rating  score  for  one  item.  The  possible  range  is  between  0  (“low”)  and 
50  (“high”). 

Scoring.  Questions  1,  2,  3  and  4  are  averaged  to  give  a  D  score  (Demand).  Questions  5,  6,  7,  8 
and  9  are  averaged  to  give  a  S  score  (Supply).  Questions  10,  11,  12  and  13  are  averaged  to 
give  a  U  score  (Understanding).  Situation  awareness  in  total  (T)  is  then  calculated  by  U  -  (D  - 
S).  Finally,  Question  14  gives  the  participant’s  confidence  in  their  ratings  of  the  above. 
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1.  Demand  on  Attentional  Resources  (capacity) 

How  demanding  were  the  tasks  on  your  attentional  resources?  Were  they  low -  high 

excessively  demanding  (high)  or  minimally  demanding  (low)? 

2.  Instability  of  Session 

How  changeable  was  the  session  (situation)?  Was  it  highly  unstable  and  low. -  high 

likely  to  change  suddenly  (high),  or  was  it  very  stable  and  straight  forward 
(low)? 

3.  Complexity  of  Session 

How  complicated  was  the  session?  Was  it  complex  with  many  interrelated  low -  high 

components  (high)  or  was  it  simple  and  straightforward  (low)? 

4.  Variability  of  Session 

How  many  variables  were  changing  in  the  session?  Were  there  a  large  low _  high 

number  of  factors  varying  (high)  or  were  there  very  few  variables  changing 
(low)? 

5.  Supply  of  Attentional  Resources  (capacity) 

How  much  of  vour  attentional  resources  were  vou  supplying  to  the  session?  lnw  high 

Were  you  making  the  greatest  possible  effort  (high)  or  giving  very  little 
attention  (low)? 

6.  Arousal 

How  aroused  were  you  in  the  session?  Were  you  alert  and  ready  for  activity  low _  high 

(high)  or  did  you  have  a  low  degree  of  alertness  (low)? 

7.  Concentration  of  Attention 

How  much  were  you  concentrating  on  the  session?  Were  you  bringing  all  low -  high 

your  thoughts  to  bear  (high)  or  was  your  attention  elsewhere  (low)? 

8.  Division  of  Attention 

How  much  was  your  attention  divided  in  the  session?  Were  yon  lnw  high 

concentrating  on  many  aspects  of  the  situation  (high)  or  focussed  on  only 
one  (low)? 

9.  Spare  Mental  Capacity 

How  much  mental  capacity  did  you  have  to  spare  in  this  session?  Did  you  low _  high 

have  sufficient  to  attend  to  many  new  variables  (high)  or  nothing  to  spare  at 
all  (low?) 

10.  Understanding  of  Session 

How  well  did  you  understand  the  session?  Did  you  understand  almost  low -  high 

everything  (high)  or  virtually  nothing  (low)? 

11.  Information  Quantity 

How  much  information  did  you  gain  from  your  environment  (inside  and  low _  high 

outside  the  cockpit)?  Did  you  receive  and  understand  a  great  deal  of 
knowledge  (high)  or  very  little  (low)? 

12.  Information  Quality 

How  good  was  the  information  you  had  gained  from  your  environment  low _  high 

(inside  and  outside  the  cockpit)?  Was  the  knowledge  communicated  very 
useful  (high)  or  was  it  of  very  little  use  (low)? 

13.  Familiarity  with  Session 

How  familiar  were  you  with  the  session?  Did  you  have  a  great  deal  of  low -  high 

relevant  experience  (high)  or  was  it  a  new  session  (low)? 

14.  Confidence  in  Ratings 

How  confident  are  vnn  of  the  ratings  vnu  have  just  made?  Are  you  verv  lnw  high 

confident  (high),  or  not  very  confident  (low)? 

Figure  2:  Situation  Awareness  Rating  Technique  Input  Form 
3. 2. 2. 4  Operator  Workload 

Another  key  determinant  of  task  performance  is  the  workload  experienced  by  the  operator 
when  engaging  in  the  tasks,  and  can  be  conceptualised  in  terms  of  both  physical  and  mental 
effort.  The  following  section  describes  a  well-validated  measure  of  workload: 

3.2.2.4.1  NASA  Task-Load  Index  (NASA-TLX) 

NASA-TLX  is  a  subjective  workload  assessment  tool  that  allows  operators  to  perform 
subjective  workload  assessments  on  operator(s)  working  with  various  human-machine 
systems.  NASA-TLX  is  a  multi-dimensional  rating  procedure  that  derives  an  overall  workload 
score  based  on  a  weighted  average  of  ratings  on  six  subscales  (see  Figure  3).  These  subscales 
include: 


•  Mental  Demands; 

•  Physical  Demands; 

•  Temporal  Demands; 

•  Own  Performance; 

•  Effort;  and. 
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•  Frustration. 

The  degree  to  which  each  of  the  six  factors  contribute  to  the  workload  of  the  specific  task  to 
be  evaluated,  from  the  operator’s  perspective,  is  determined  by  their  responses  to  pair-wise 
comparisons  among  the  six  factors.  Magnitude  ratings  on  each  subscale  are  obtained  after 
each  performance  of  a  task  or  task  segment.  Ratings  of  factors  deemed  most  important  in 
creating  the  workload  of  a  task  are  given  more  weight  in  computing  the  overall  workload 
score,  thereby  enhancing  the  sensitivity  of  the  scale. 


Figure  3:  NASA  TLX  Dimensions  and  Scales 

The  simple  nature  of  the  scale  permits  subjects  to  provide  ratings  quickly  in  any  operational 
settings.  The  scale  can  be  administered  using  paper  or  via  a  direct  operator  input  program,  in 
real-time  or  administered  retroactively,  and  it  has  been  demonstrated  that  little  information  is 
lost  when  ratings  are  given  retrospectively;  a  high  correlation  was  found  between  ratings 
obtained  “on-line”  and  those  obtained  retrospectively.  The  Task  Load  Index  has  been  tested  in 
a  variety  of  experimental  tasks  that  range  from  simulated  flight  to  supervisory  control 
simulations  and  laboratory  tasks  (e.g.,  the  Sternberg  memory  task,  choice  reaction  time, 
critical  instability  tracking,  compensatory  tracking,  mental  arithmetic,  mental  rotation,  target 
acquisition,  grammatical  reasoning,  etc.).  NASA-TLX  can  be  used  to  assess  workload  in 
various  human-machine  environments  such  as  aircraft  cockpits;  C2  workstations;  supervisory 
and  process  control  environments;  simulations  and  laboratory  tests. 
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3. 2. 2. 5  Opera  tor  Fa  tigue 

Operator  performance,  as  well  as  general  acceptance  of  the  system,  will  be  significantly 
affected  by  operational  fatigue.  It  is  therefore  important  to  consider  the  effects  of  operator 
fatigue  on  the  system  in  order  to  avoid  negative  outcomes  due  to  fatigue-induced  errors  (e.g., 
when  monitoring  an  automated  system  over  extended  periods  of  time). 

The  Fatigue  rating  questionnaire  provides  a  simple  subject  self  report  measure  of  fatigue  that 
can  be  administered  at  the  same  time  as  other  system  performance  or  acceptance  measures 
(see  Figure  4). 


FATIGUE  RATING 

(Circle  the  number  of  the  statement  which  describes  how 
you  feel  RIGHT  NOW.) 

1 

Fully  Alert;  Wide  Awake;  Extremely  Peppy 

2 

Very  Lively;  Responsive;  But  Not  at  Peak 

3 

Okay;  Somewhat  Fresh 

4 

A  Little  Tired;  Less  Than  Fresh 

5 

Moderately  Tired;  Let  Down 

6 

Extremely  Tired;  Very  Difficult  to  Concentrate 

7 

Completely  Exhausted;  Unable  to  Function 

Effectively;  Ready  to  Drop 

Comments: 

Figure  4:  Fatigue  Questionnaire 
3. 2.2.6  Operator  Acceptance 

The  Technology  Acceptance  Model  (TAM)  is  an  information  systems  theory  that  models  how 
operators  come  to  accept  and  use  a  technology.  The  model  suggests  that  when  operators  are 
presented  with  a  system,  a  number  of  factors  will  influence  their  decision  about  how  and 
when  they  will  use  it,  notably: 

•  Perceived  Usefulness  (PU):  This  is  defined  as  ‘the  degree  to  which  a  person  believes 
that  using  a  particular  system  would  enhance  his  or  her  job  performance’. 

•  Perceived  Ease-of-Use  (PEOU):  This  is  defined  as  ‘the  degree  to  which  a  person 
believes  that  using  a  particular  system  would  be  free  from  effort.’ 

The  TAM  utilizes  a  questionnaire  that  has  been  assessed  for  robustness  across  populations 
and  predictive  validity.  Studies  have  found  high  reliability  and  good  test-retest  reliability  and 
have  found  that  the  instrument  had  predictive  validity  for  intent  to  use,  self-reported  usage 
and  attitude  toward  use.  The  sum  of  this  research  has  confirmed  the  validity  of  the  instrument, 
and  supports  its  use  with  different  populations  of  operators  and  different  software  choices. 
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3. 2. 2. 6.1  Assessments  of  System  Usability  and  Usefulness 

A  questionnaire  relating  to  the  high-level  usability  aspects  of  the  OM1  (e.g.,  suitability  of 
screen  windows,  keyboard  and  joystick)  is  used  to  assess  operator  perceptions  of  system 
usability.  Ratings  are  based  on  a  five -point  Likert  scale;  ranging  from  1 :  Strongly  Disagree  to 
5:  Strongly  agree.  For  example: 


Statement 

© 

Strongly 

Disagree 

Disagree 

Border 

Agree 

© 

Strongly 

Agree 

Suggested 

Improvements 

1 .  The  size  of  the  .... 
Window  is  suitable. 

□ 

□ 

□ 

□ 

□ 

A  questionnaire  relating  to  the  perceived  utility  of  the  workstation  functions  and  capabilities 
(e.g.,  drill-down)  is  used  to  assess  operator  perceptions  of  system  usefulness.  Ratings  are 
based  on  a  five-point  Likert  scale;  ranging  from  1:  Strongly  Disagree  to  5:  Strongly  agree.  For 
example: 


Statement 

‘It  is  USEFUL  to  be  able 
to...’ 

© 

Strongly 

Disagree 

Disagree 

Border 

Agree 

© 

Strongly 

Agree 

Suggested 

Improvements 

1 .  Drill-down  to  find 
out  more  information. 

□ 

□ 

□ 

□ 

□ 

3. 2. 2. 7  Operator  Trust 

Trust  can  be  defined  as  the  extent  to  which  an  operator  is  confident  in  and  willing  to  act  on 
the  basis  of  the  recommendations,  actions,  and  decisions  of  an  artificially  intelligent  decision 
aid.  Flowever,  trust  is  not  a  simple  uni-dimensional  variable.  It  is  possible  to  be  correctly 
distrusting  of  a  system  (e.g.,  when  it  is  unreliable),  but  also  to  be  too  trusting  (‘over-trusting’ 
or  complacent)  or  not  trusting  enough  (‘under-trusting’  or  sceptical). 

Operator  trust  and  acceptance  of  automation  is  determined  from  the  outcome  of  a  comparison 
process  between  the  perceived  reliability  of  the  automated  aid  (i.e.,  trust  in  aid)  and  the 
perceived  reliability  of  manual  control  (i.e.,  trust  in  self).  Decision  making  quality  will 
increase  when  the  operator  is  able  to  compare  the  abilities  of  the  decision  aid  with  their  own 
abilities.  A  subjective  assessment  can  be  used  to  measure  the  degree  of  operator  trust  in  the 
decision  aid.  Overall  trust  in  the  decision  aid  is  determined  by  cognition-based  trust  (i.e.,  trust 
relating  to  the  operator’s  perception  of  the  automation)  and  affect-based  trust  (i.e.,  trust 
relating  to  the  operator’s  emotive  response  to  automation).  Three  factors  underpin  cognition- 
based  trust  (i.e.,  perceived  understandability,  technical  competence,  and  reliability  [of  the 
system]),  and  two  factors  undeipin  affect-based  trust  (i.e.,  faith  [in  the  system]  and  personal 
attachment  [to  the  system]).  Each  of  these  five  factors  has  five  sub-items  as  shown  in  Figure 
5. 
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1 .  Perceived  reliability 

R1.  The  system  always  provides  the  advice  I  require  to  make  my  decision. 

R2.  The  system  performs  reliably. 

R3.  The  system  responds  the  same  way  under  the  same  conditions  at  different  times. 

R4.  can  rely  on  the  system  to  function  properly. 

R5.  The  system  analyzes  problems  consistently. 

2.  Perceived  technical  competence 

T1.  The  system  uses  appropriate  methods  to  reach  decisions. 

T2.  The  system  has  sound  knowledge  about  this  type  of  problem  built  into  it. 

T3.  The  advice  the  system  produces  is  as  good  as  that  which  a  highly  competent 
person  could  produce. 

T4.  The  system  correctly  uses  the  information  I  enter. 

T5.  The  system  makes  use  of  all  the  knowledge  and  information  available  to  it  to 
produce  its  solution  to  the  problem. 

3.  Perceived  understandability 

U 1 .  I  know  what  will  happen  the  next  time  I  use  the  system  because  I  understand 
how  it  behaves. 

U2.  I  understand  how  the  system  will  assist  me  with  decisions  I  have  to  make. 

U3.  Although  I  may  not  know  exactly  how  the  system  works,  I  know  how  to  use  it  to 
make  decisions  about  the  problem. 

U4.  It  is  easy  to  follow  what  the  system  does. 

U5.  I  recognize  what  I  should  do  to  get  the  advice  I  need  from  the  system  the  next 
time  I  use  it. 

4.  Faith 

FI .  I  believe  advice  from  the  system  even  when  I  don’t  know  for  certain  that  it  is 
correct. 

F2.  When  I  am  uncertain  about  a  decision  I  believe  the  system  rather  than  myself. 

F3.  If  I  am  not  sure  about  a  decision,  I  have  faith  that  the  system  will  provide  the  best 
solution. 

F4.  When  the  system  gives  unusual  advice  I  am  confident  that  the  advice  is  correct. 

F5.  Even  if  I  have  no  reason  to  expect  the  system  will  be  able  to  solve  a  difficult 
problem,  I  still  feel  certain  that  it  will. 

5.  Personal  attachment 

PI .  I  would  feel  a  sense  of  loss  if  the  system  was  unavailable  and  I  could  no  longer 
use  it. 

P2.  I  feel  a  sense  of  attachment  to  using  the  system. 

P3.  I  find  the  system  suitable  to  my  style  of  decision-making. 

P4.  I  like  using  the  system  for  decision-making. 

P5.  I  have  a  personal  preference  for  making  decisions  with  the  system. 

Figure  5:  Human-Computer  Trust  rating  scale  (Madsen  and  Gregor,  2000) 

3.3  ESS  OMI  Design  Guidelines 


3.3.1  General  Design  Goals 


3.3.1. 1 

Employ  Operator-Centered  Principles 

Source:  HFDS,  2003:  Sheridan,  2000: 
DefStan,  1996,  AHCI,  1998,  DISA, 

1996;  MS1472F;  Endsleyand  Kaber, 
1999. 

Guideline(s): 

1. 

The  ESS  shall  be  used  to  support  the  operator(s)  where  appropriate  (e.g.,  human-centered 
ESS),  not  implemented  simply  because  the  technology  is  available  (e.g.,  technology- 
centered  ESS). 

2. 

The  ESS  shall  be  design  to  match  the  operator’s  mental  model  of  the  domain  as  well  as 
the  processes  underlying  system  operation. 

3. 

The  ESS  shall  help  or  enable  the  operators  to  carry  out  their  responsibilities  and  tasks 
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safely,  efficiently,  and  effectively.  [Carrying  out  a  task  effectively  means  producing  the 
desired  result.  Carrying  out  a  task  efficiently  means  that  the  desired  result  is  produced  with 
a  minimum  of  waste  (usually  in  relation  to  time)]. 

4.  The  operator  shall  always  have  final  authority  over  the  allocation  of  ESS  functions  (i.e., 
task  allocated  to  human  and/or  system). 

5.  Functions  shall  be  automated  only  to  attain  greater  overall  effectiveness,  efficiency, 
reliability,  simplicity,  economy,  and  system  safety  without  reducing  human  involvement, 
situation  awareness,  or  human  performance  in  carrying  out  the  intended  task. 

6.  The  relationships  between  display,  control,  decision  aid,  and  information  structure  and 
operator  tasks  and  functions  shall  be  clear  to  the  operator. 


Discussion: 

2.  The  operator  must  interpret  the  information  provided  to  him/her.  The  operator’s  training, 
experience,  biases  will  influence  the  quality  of  the  decision  and  execution  of  their  task  (s). 

4.  The  reasoning  behind  this  guideline  is  twofold.  First,  it  is  ultimately  the  operator  who  is 
responsible  for  the  task.  Second,  ESS  automation  is  subject  to  failure.  Therefore,  it  is  the 
operator,  not  the  automation  who  must  be  in  authority  of  the  system  with  the  automation 
playing  a  subservient  role. 

5.  Automation  can  lead  to  distraction  from  the  primary  task,  increased  workload,  boredom  or 
complacency. 


6.  The  operator  needs  to  be  able  to  see  clearly  how  the  display  or  decision  aid,  and  so  on, 
facilitates  the  completion  of  the  necessary  task 


Evaluation  Measures  and  Methodoloqv: 

Design: 

Inspect 

Demonstrate 

Experiment 

•  Analysis  and  design  methodology  compliant  with 
MIL-HDBK-46855A  (Human  Engineering 

Program  Process  and  Procedures) 

0 

□ 

□ 

•  Objective  measures  of  the  adequacy  of  OMI 
usability  through  usability  testing 

□ 

□ 

0 

Operator  Acceptance: 

•  Subjective  assessment  of  the  adequacy  of 
system  usability  and  utility  (in  terms  of  its 
design)  using  ‘walk-through’  or  heuristic 
analysis1  (e.g.,  Nielsen,  1994). 

□ 

0 

□ 

•  Questionnaire-based  assessments  of  the 
adequacy  of  operator’s  perceptions  of  system 

□ 

□ 

0 

1  The  criteria  to  make  such  a  judgment  will  relate  to  all  aspects  of  the  OMI  design.  This  would  mean 
that  most  of  the  criteria  within  the  rest  of  the  document  would  need  to  have  been  met. 
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usability  and  utility 

Relationship  to  Other  Guidelines: 

•  3. 3. 1.2  Optimize  Human-System  Interaction 

•  3.3.2.  Employ  Operator-centered  OMI  Design 

•  3. 4.2. 3  Keep  Operators  in  Control 

•  3.5.1  Adopt  Operator-Centered  Design  Principles 


3.3.1. 2 


Optimize  Human-System  Interaction 


Source:  HFDS,  2003;  DISA,  1996; 
Parasuraman  and  Riley,  1997;  AHCI, 
1998;  MS1472F;  Endsley,  1996; 
DefStan,  1996;  Endsley  and  Kaber, 


1999. 


Guideline(s): 

1.  The  ESS  shall  provide  sufficient  information  to  keep  the  operator  informed  of  its  operating 
mode,  intent,  function,  and  output;  inform  the  operator  of  system  failure  or  degradation; 
inform  the  operator  if  potentially  unsafe  modes  are  manually  selected;  do  not  interfere  with 
manual  task  performance;  and  allow  for  manual  override. 

2.  ESS  system  functioning  shall  be  transparent  to  the  operator  at  all  times. 

3.  The  operator  shall  have  active  involvement  in  the  operation  of  the  ESS.  Operators  shall  be 
given  an  active  role  through  relevant  and  meaningful  tasks  in  the  operation  of  a  system, 
including  when  the  tasks  are  automated. 

4.  ESS  functionality  shall  be  appropriate  to  the  operator’s  level  of  expertise  with  the  system 
(e.g.,  shortcuts  such  as  function  keys  can  be  provided  for  the  more  experienced 
operators). 

5.  ESS  functioning  shall  not  increase  the  demands  for  cognitive  resources  (thinking  or 
conscious  mental  processes). 

6.  Extreme  levels  of  workload  (low  or  high)  due  to  ESS  functioning  shall  be  avoided  (to 
maximize  operator-in-the-loop  and  reduce  automation  bias). 

7.  Operator  interaction  with  the  ESS  shall  not  require  the  operator  to  take  significant  amounts 
of  attention  away  from  the  primary  task. 

8.  ESS  shall  not  interrupt  at  inappropriate  times  such  as  during  periods  of  high  workload  or 
during  critical  moments  in  a  process. 

9.  An  ESS  task  shall  be  less  difficult  to  carry  out  than  the  manual  task  it  replaces. 

10.  Data  that  are  needed  by  the  operator  shall  be  easily  accessible. 


1 1 .  The  ESS  shall  allow  the  operator  to  interact  directly  with  objects  which  are  important  to  the 
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operator’s  tasks. 


Discussion: 

2.  The  transparency  of  system  functioning  (i.e.,  the  properties  of  the  ESS  which  allow  the 
operator  to  understand  its  actions)  will  increase  the  predictability  of  the  ESS  (e.g.,  reliability 
of  automatically  detecting  and  prioritizing  tracks)  by  ensuring  the  operator  is  cognisant  of 
the  limitations  of  the  ESS.  In  addition,  it  is  very  important  that  operators  understand  why, 
and  under  what  conditions,  the  ESS  might  make  errors.  Trust  should  increase  if  operators 
receive  informative  feedback  in  the  event  of  an  ESS  error  (e.g.,  explanation  of  system 
error).  For  general  information  on  providing  feedback  to  the  operator,  see  MIL-STD  1472F. 

3.  Operator  awareness  of  ESS  state  can  not  be  sustained  passively.  Active  involvement  is 
essential  for  operators  to  exercise  their  responsibilities  and  be  able  to  respond  to 
emergencies.  Reducing  active  involvement  may  be  detrimental  to  the  operator’s 
understanding  of  important  information,  may  lead  to  longer  response  times  in  case  of 
emergencies,  or,  in  the  long  term,  may  lead  to  loss  of  relevant  knowledge  or  skills. 

4.  ESS  functions  that  increase  the  demand  for  cognitive  resources  of  the  operator  is  an 
artefact  of  poor  design.  Expert  operators  in  complex,  dynamic  systems  have  been 
observed  to  cope  with  poorly  designed  ESSs  by  using  only  a  subset  of  the  available 
functionality,  especially  during  periods  of  high  workload. 

6.  ESSs  can  cause  extreme  workload  levels  by  increasing  workload  when  it  is  already  high 
and  decreasing  workloads  that  are  already  low.  ESSs  are  often  introduced  to  reduce 
workload.  However,  reduction  of  workload  may  not  always  be  advantageous,  for  example, 
if  workload  is  already  low. 

7.  When  an  ESS  requires  the  operator  to  devote  a  significant  amount  of  attention  to  adjusting 
or  monitoring  the  ESS  functioning,  this  removes  the  operator  away  from  minute-to-minute 
operations,  thereby  taking  the  operator  out  of  the  loop.  This  can  be  especially  dangerous  if 
an  abnormal  situation  occurs  that  needs  to  be  remedied  quickly. 

8.  An  interruption  during  high  workload  or  at  a  critical  moment  can  cause  a  delay  in  the 
operator’s  ability  to  respond  to  an  ESS  malfunction,  leading  to  a  potential  failure.  If  the 
operator  is  attending  to  an  ESS  malfunction  and  is  interrupted,  the  interruption  depletes 
the  operator’s  mental  resources  causing  him  to  be  less  capable  of  averting  the  potential 
failure. 


10.  Operator  requirements  can  serve  as  a  guide  to  whether  the  data  are  available  at  all  times, 
accessible  at  the  operators’  discretion,  or  not  at  all  if  the  operator  does  not  need 
information. 


Evaluation  Measures  and  Methodoloqies: 

Inspect 

Demonstrate 

Experiment 

Design: 

•  Analysis  and  design  methodology  compliant  with 
MIL-HDBK-46855A  (Human  Engineering 

Program  Process  and  Procedures) 

0 

□ 

□ 

•  Objective  measures  of  the  adequacy  of  OMI 
ease  of  use  and  usefulness  through  Usability 
testing  (e.g.,  observation-based  studies) 

□ 

□ 

0 
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Operator  Situation  Awareness: 

•  Subjective  assessment  (i.e. ,  questionnaire)  of 
the  adequacy  of  the  operator’s  Situation 
Awareness  of  ESS  functioning. 

□ 

□ 

0 

•  Subjective  (i.e.,  questionnaire)  and  objective 
(e.g.,  performance,  errors)  assessment  of  the 
adequacy  of  the  operator’s  understanding  of  the 
limitations  of  the  ESS. 

□ 

□ 

0 

•  Subjective  assessment  (i.e.,  questionnaire)  of 
the  adequacy  of  the  operator’s  Situation 
Awareness  of  task-relevant  objects  in  the 
environment. 

□ 

□ 

0 

Operator  Workload 

•  Subjective  assessment  (i.e.,  questionnaire)  of 

the  adequacy  of  the  operator’s  workload. 

□ 

□ 

0 

Relationship  to  Other  Guidelines: 


•  3. 3. 1.1  Employ  Operator-Centered  Principles 

•  3. 4. 2. 4  Maximize  Operator  Situation  Awareness  by  Increasing  System  Transparency 

•  3.4.3. 1  Keep  Operators ‘In-the-Loop’ 


3.3.1. 3 

Promote  ESS  Robustness  and  Resilience  to 

Source:  HFDS,  2003:  MS1472F:  DISA, 

Operator  Error 

1996;  Sheridan,  2000;  AHCI,  1998; 
Parasuraman  and  Riley,  1997; 

DefStan,  1996. 

Guideline(s): 


1 .  The  ESS  shall  be  resistant  to  operator  error  while  tolerating  some  reasonable  level  of 
“error”  and  response  variability. 

2.  The  ESS  shall  be  able  to  monitor  operator  interactions  and  to  warn  of  operator  errors. 

3.  ESS  functions  shall  be  capable  of  being  overridden  by  the  operator  in  an  emergency.  ESS 
functioning  shall  not  be  difficult  or  time  consuming  to  turn  on  or  off. 

4.  Operators  shall  not  be  too  reliant  on  ESS  functioning  to  the  extent  that  their  skills  are 
degraded  by  extended  use  of  the  ESS  and  that  they  can  no  longer  safely  recover  from 
emergencies  or  operate  the  ESS  manually  if  the  ESS  fails. 

5.  ESS  shall  not  be  able  to  veto  operator  actions  leaving  the  operator  without  means  to 
override  or  violate  the  rules  that  govern  the  ESS  unless  there  is  not  enough  time  for  the 
operator  to  make  a  decision. 
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6.  ESS  interactivity  (i.e.,  navigation,  functionality,  features,  information  structure)  shall  be 
consistent  within  and  between  systems. 

7.  ESS  status  during  system  setup  shall  be  transparent  to  the  operator  (e.g.,  system  failure 
due  to  system  setup  or  manual  input  of  information). 

8.  Allocation  of  tasks  shall  be  flexible  and  adaptable  (e.g.,  a  task  allocated  to  an  ESS  or  an 
operator  can  be  adapted  according  to  the  context)  and  the  operator  shall  always  have 
authority  over  how  the  tasks  are  allocated. 

9.  The  ESS  shall  make  it  clear  whether  the  operator  or  the  ESS  is  supposed  to  perform  a 
particular  task  at  a  specific  time. 

10.  The  allocation  of  tasks  related  to  decision/action  to  the  ESS  shall  be  under  the  authority  of 
the  operator;  particularly  under  situations  of  greater  uncertainty  and  risk. 

1 1 .  To  increase  operator  trust  in  the  ESS,  ESS  performance  shall  be:  reliable  and  predictable 
with  minimal  errors;  robust  (able  to  perform  under  a  variety  of  circumstances);  familiar  (use 
terms  and  procedures  familiar  to  the  operator);  and,  useful. 

12.  The  ESS  shall  be  available  to  the  operator  as  needed. 

13.  The  ESS  shall  not  interfere  with  task  performance. 

14.  The  ESS  shall  provide  accurate  and  reliable  information. 


Discussion: 

1 .  To  make  a  system  error  resistant  is  to  make  it  difficult  for  an  operator  to  make  an  error. 
Simplicity  in  design  and  the  provision  of  clear  information  are  tools  to  improve  error 
resistance.  Electronic  checklists  also  have  the  potential  to  improve  error  resistance  by 
providing  reminders  of  items  that  need  to  be  completed.  Error  tolerance  is  the  ability  to 
mitigate  the  effects  of  human  errors  that  are  committed.  Error  tolerance  can  be  improved 
by  adding  monitoring  capabilities  to  the  ESS.  Acceptable  levels  of  error  and  response 
variability  enhance  learning  (of  the  operator). 

4.  A  balance  is  needed  between  the  efficiency  created  by  the  ESS,  and  the  need  for  the 
operator  to  be  able  to  recover  from  emergencies  and  control  the  ESS  manually  in  case  the 
ESS  fails. 

5.  The  resumption  of  manual  control  needs  to  be  within  the  capacity  of  the  operator,  without 
relying  on  manual  skills. 

6.  There  are  many  possible  types  of  interaction,  such  as  menu  selection,  direct  manipulation, 
and  form-filling.  An  example  of  inconsistent  interaction  would  be  having  one  ESS  require 
filling  in  forms  as  the  interaction  method,  whereas  another  ESS  requires  menu-driven 
interaction.  However,  in  the  case  of  repetitive  movement  using  a  single  input  device  (e.g., 
track-ball),  operator  fatigue  can  be  mitigated  by  using  an  alternative  method  of  interaction 
(e.g.,  touch  screen  in  addition  to  the  track-ball). 

7.  ESS  failures  are  often  due  to  setup  error.  Although  the  ESS  itself  could  check  some  of  the 
setup,  independent  error-checking  equipment  or  procedures  may  be  needed.  The  operator 
needs  to  be  able  to  distinguish  whether  a  failure  occurred  due  to  the  ESS  setup  or  due  to 

_ an  inaccuracy  in  the  input  information.  A  failure  could  have  been  caused  by  a  malfunction 
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of  an  algorithm  or  by  the  input  of  inaccurate  data.  ESS  operations  that  are  easily 
interpretable  or  understandable  by  the  operator  can  facilitate  the  detection  of  improper 
operation  and  the  diagnosis  of  malfunctions. 

8.  Problems  with  an  ESS  can  occur  when  ESS-generated  options  do  not  apply  to  a  situation 
and  the  operator  is  restricted  to  the  options  provided  by  the  ESS. 

10.  High  levels  of  ESS  automation  (i.e.,  the  ESS  initiates  and  performs  the  task)  can  be  used 
for  tasks  involving  relatively  little  uncertainty  and  risk.  Since  the  decision  as  to  whether  or 
not  a  situation  is  one  of  greater  uncertainty  and  risk  will  be  made  by  the  operator, 
allocation  should  always  be  under  control  of  the  operator. 

1 1 .  T rust  in  automation  tends  to  be  relatively  stable.  However,  changes  in  trust  may  occur  over 
time.  Operator  trust  in  automation  can  increase  with  reliable  and  predictable  performance. 
Decreases  in  trust  may  occur  as  a  result  of  some  critical  error  or  automation  failure.  It  is 
more  difficult  for  operators  to  regain  trust  in  automation  after  a  failure  than  to  develop  an 
initial  trust.  Higher  trust  in  automation  is  not  always  better  because  automation  errors  may 
be  overlooked  due  to  complacency.  Decreases  in  trust  typically  occur  suddenly,  but 
increases  happen  slowly  and  steadily.  The  consequences  of  an  automation  failure  (for 
example,  the  magnitude  of  an  error)  impact  the  decline  in  trust. 

13.  An  operator  will  be  less  likely  to  accept  an  automated  system  that  interferes  with  their 
ability  to  perform  tasks. 

14.  When  operators  believe  the  system  to  be  highly  reliable,  they  place  greater  trust  in  it. 
However,  there  is  a  trade-off  involved  with  a  constant  high  level  of  automation  reliability 
and  predictability.  Constant  high  levels  of  reliability  and  predictability  may  be  more  likely  to 
promote  complacency  and  may  cause  operators  to  monitor  the  system  with  less  vigilance. 


Evaluation  Measures  and  Methodoloqies: 

Design: 

Inspect 

Demonstrate 

Experiment 

•  Assess  analysis  and  design  methodology 
compliant  with  MIL-HDBK-46855A  (Human 
Engineering  Program  Process  and  Procedures) 

0 

□ 

□ 

•  Objective  measures  of  the  adequacy  of  OMI 
ease  of  use  and  usefulness  through  Usability 
testing  (e.g.,  observation-based  studies) 

System  Performance: 

□ 

□ 

0 

•  Predictability  of  the  system 

Operator  Performance: 

□ 

0 

□ 

•  Periodic  objective  assessment  of  operator  skill- 
fade  (e.g.,  assess  competence  to  perform  task 
manually)  by  a  Human  Factors  practitioner.  For 
example,  at  recurrent  training  intervals. 

Operator  Situation  Awareness: 

□ 

□ 

0 

•  Subjective  assessment  (i.e.,  questionnaire)  of 

□ 

□ 

0 
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the  adequacy  of  the  operator’s  Situation 
Awareness  of  system  functioning. 

•  Subjective  (i.e.,  questionnaire)  and  objective 
(e.g.,  performance  errors)  assessment  of  the 
adequacy  of  the  operator’s  understanding  of  the 
limitations  of  the  system. 

□ 

□ 

0 

•  Subjective  assessment  (i.e.,  questionnaire)  of 
the  adequacy  of  the  operator’s  Situation 
Awareness  of  task-relevant  objects  in  the 
environment. 

□ 

□ 

0 

Relationship  to  Other  Guidelines: 


•  3.3.4  Provide  System  Response  and  Feedback 

•  3.3.5  Support  Identification  and  Management  of  ESS  Faults  and  Failures 

•  3.4.3. 1  Keep  Operators  in  Control 

•  3. 4.2. 4  Maximize  Operator  Situation  Awareness  by  Increasing  System  Transparency 


3.3.1. 4 

Support  Operator  Monitoring  of  ESS 

Source:  HFDS,  2003;  Parasuraman 

Functioning 

and  Riley,  1997;  Sheridan,  2000; 

Endsley  and  Kaber,  1997. 

Guideline(s): 


1 .  Informative  feedback  shall  be  given  in  case  of  an  ESS  failure;  such  as  the  likely  cause 
and/or  location  of  the  failure. 

2.  The  ESS  shall  be  designed  so  that  operators  are  able  to  monitor  the  ESS  and  the 
functionality  of  its  hardware  and  software,  including  the  display  of  status  and  trend 
information,  as  needed. 

3.  ESS  tasks  shall  be  designed  so  that  operators  are  involved  in  active  control  and 
monitoring  rather  than  just  passive  monitors. 

4.  System  designers  shall  allow  adequate  cognitive  resources  for  monitoring  of  the  ESS  by 
ensuring  that  task  load  does  not  become  excessive. 

5.  Operators  shall  not  be  required  to  perform  purely  monitoring  tasks  for  longer  than  20 
minutes  at  a  time. 

6.  Important  events  shall  occur  in  the  same  location  on  a  display  in  order  to  promote  effective 
monitoring  performance,  including  when  operators  must  monitor  multiple  displays. 

7.  The  ESS  shall  provide  some  type  of  indication  the  system  is  still  being  monitored  by  some 
automatic  system. 


31 


8.  Critical  ESS  functions  shall  be  independently  monitored  by  the  operator.  A  critical  function 
is  a  function  that  can  cause  system  failure  when  a  malfunction  is  not  attended  to 
immediately. 

9.  Operators  shall  be  given  an  adequate  understanding  (mental  model)  of  how  the  ESS 
works  in  order  to  monitor  effectively. 

1 0.  Intermittent  periods  of  task  monitoring  by  the  operator  shall  be  used  during  extended 
periods  of  task  automation  to  improve  monitoring  of  the  ESS. 

1 1 .  The  effects  on  vigilance  due  to  the  use  of  ESS  shall  be  considered  before  automating 
tasks  or  functions. 

12.  The  ESS  shall  behave  predictably  so  that  the  operator  knows  the  purpose  of  the  ESS 
functioning  and  how  the  task  will  be  affected  by  that  functioning. 

13.  The  ESS  shall  provide  means  to  indicate  to  the  operator  that  data  are  missing,  incomplete, 
unreliable,  or  invalid  or  that  the  system  is  relying  on  backup  data. 


Discussion: 

1 .  Different  messages  shall  be  given,  depending  on  whether  the  error  is  due  to  the  central 
system  or  whether  the  error  is  local. 

2.  One  way  that  this  can  be  accomplished  is  by  providing  the  operator  with  access  to  raw 
data  that  the  ESS  processes. 

3.  Failures  in  ESS  functioning  may  be  easier  to  detect  when  operators  are  involved  in  both 
active  control  and  monitoring,  than  when  they  are  just  passive  monitors. 

4.  Operators  using  ESS  may  experience  higher  levels  of  mental  workload  than  manual 
controllers  due  to  monitoring,  diagnosis,  and  planning,  with  significant  cognitive  demand 
resulting  from  relatively  “simple”  vigilance  tasks. 

5.  Operators  may  become  complacent  in  monitoring  an  ESS  if  they  have  other  tasks  to 
complete  simultaneously.  Such  decrements  in  operator  monitoring  of  an  ESS  have  been 
observed  to  occur  in  the  laboratory  in  as  little  as  20  minutes. 

6.  Operators  will  be  able  to  detect  a  particular  event  more  easily  if  they  know  where  that 
event  will  occur  (i.e. ,  spatial  certainty).  Spatial  uncertainty  has  been  shown  to  increase 
perceived  workload  and  decrease  performance  efficiency.  If  operators  do  not  know  where 
on  a  display  an  event  will  occur  then  they  must  engage  in  visual  scanning  to  look  for  the 
event. 

8.  When  a  function  is  critical,  combining  the  monitoring  of  that  critical  function  with  other, 
possibly  less  critical  functions  may  lead  to  delays  in  response.  When  a  critical  function  is 
independently  monitored,  an  operator  can  respond  to  a  malfunction  very  quickly  (within 
one  second).  If  an  operator  is  attending  to  another  task  when  there  is  a  malfunction,  there 
will  be  a  delay  in  the  operator’s  response  (several  seconds).  In  this  period  of  delayed 
response,  the  malfunction  can  cause  the  system  to  fail.  For  this  reason,  critical  functions 
require  constant  attention.  Critical  ESS  functions  do  assist  in  the  completion  of  critical 
tasks,  however  they  do  not  assist  in  freeing  the  operator  to  attend  to  other  tasks. 


32 


9.  Operators  must  possess  accurate  mental  models  of  the  ESS  in  order  to  monitor  effectively, 
comprehend  current  situations,  plan  their  actions,  predict  future  system  states,  remember 
past  instructions,  and  diagnose  system  failures.  One  way  to  establish  adequate  mental 
models  is  through  training. 

1 0.  Complacency  is  a  major  concern  with  the  automation  of  tasks.  If  practicable  (i.e.,  the 
operator  is  able  to  perform  the  task(s)  manually),  intermittent  periods  of  manual  control 
have  been  advocated  as  a  means  of  minimizing  complacency.  Decrement  of  cognitive 
abilities  such  as  Situation  Awareness  and  the  loss  of  manual  skills  may  also  occur,  making 
transitions  from  automated  to  manual  systems  difficult.  Because  automation  of  tasks  can 
decrease  basic  manual  skills,  these  skills  should  be  used  and  maintained,  if  practicable. 
Intermittent  periods  of  manual  control  during  which  ESS  functioning  is  suspended 
periodically  can  promote  optimal  operator  performance,  and  allow  better  recovery  from 
failure,  regardless  of  the  type  of  task  that  is  automated. 

1 1 .  A  vigilance  decrement,  that  is,  a  continuously  decreasing  ability  to  maintain  attention  over 
time  while  monitoring,  may  occur  with  the  automation  of  tasks.  Vigilance  decrements  do 
not  occur  because  monitoring  tasks  are  under-stimulating.  Rather,  they  require  a  large 
amount  of  cognitive  resources  and  are  often  frustrating.  Vigilance  decrements  have  been 
observed  to  occur  for  both  expert  and  novice.  How  hard  the  operator  must  work  in  order  to 
maintain  vigilance  can  be  determined  by  at  least  two  factors.  First,  workload  is  affected  by 
the  ease  with  which  relevant  signals  can  be  detected.  Signals  that  have  low  salience  are 
more  difficult  to  detect  than  signals  high  in  salience.  Visual  fatigue  will  also  require  more 
effort  to  be  expended  in  order  to  detect  a  signal.  Second,  musculo-skeletal  fatigue 
associated  with  maintaining  a  fixed  posture  will  increase  the  workload  needed  to  perform 
optimal  monitoring. 


12.  The  predictability  of  ESS  functioning  allows  the  operator  to  know  what  to  expect  when  the 
ESS  is  functioning  correctly.  This  makes  it  easier  for  the  operator  to  recognize  when  the 
ESS  is  not  functioning. 


Evaluation  Measures  and  Methodoloqies: 

Operator  Performance: 

Inspect 

Demonstrate 

Experiment 

•  Objective  assessment  of  the  adequacy  of 

operator’s  accurate  and  timely  detection  of  ESS 
faults  and  failures. 

□ 

□ 

0 

Operator  Situation  Awareness: 

•  Subjective  assessment  (i.e.,  questionnaire)  of 
the  adequacy  of  operator  Situation  Awareness 
of  system  functioning. 

□ 

□ 

0 

Operator  Workload: 

•  Subjective  assessment  (i.e.,  questionnaire)  of 
the  adequacy  of  operator  mental  and  physical 
workload. 

□ 

□ 

0 

Operator  Fatigue: 

□ 

□ 

0 

•  Subjective  assessment  (i.e.,  questionnaire)  of 
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the  adequacy  of  operator  fatigue. 

Operator  Trust: 

•  Subjective  assessment  (i.e. ,  questionnaire)  of 
the  adequacy  of  Operator  trust 


Relationship  to  Other  Guidelines: 

•  3. 3. 1.1  Employ  Operator-Centered  Principles 

•  3.3.4  Provide  System  Response  and  Feedback 

•  3. 4. 2. 4  Maximize  Operator  Situation  Awareness  by  Increasing  System  Transparency 


•  3.6.2  Train  to  Overcome  ‘Automation  Biases’ 


1 _ 1 

3.3.2 

Employ  Operator-centered  OMI  Design 

Source:  HFDS,  2003;  Nielsen,  1994; 
Hutchins  et  al,  1999;  Zachary  and 

Ryder,  1997;  MS1472F;  AHCI,  1998; 
DefStan,  1996. 

Guideline(s): 


1.  The  ESS  and  associated  integrated  information  displays  shall  be  intuitive,  easy  to 
understand,  and  easy  to  use. 

2.  The  ESS  shall  be  simple  for  the  operators  to  learn. 

3.  Support  the  operator  recognising  objects,  actions  and  options  rather  than  relying  on  the 
operator’s  memory  (recall). 

4.  The  ESS  interface  shall  represent  the  simplest  design  consistent  with  functions  and  tasks 
of  the  operator. 

5.  The  ESS  interface  shall  be  consistent  with  the  expectations  and  understandings  of  the 
operator  and  shall  reflect  an  obvious  logic  based  on  operator  task  needs  and  capabilities. 

6.  Navigation  aids  (e.g.,  landmarks)  shall  make  it  easy  for  operators  to  know  where  they  are 
in  the  data  space. 

7.  The  OMI  layout  shall  be  organized  according  to  the  human  perceptual  system  to  reduce 
the  operator’s  workload  (e.g.,  proximity,  matching  patterns,  unity,  continuity,  balance 
principles). 

8.  Where  possible,  spatial  representations  of  information  shall  be  used  instead  of  verbal  or 
textual  displays  to  reduce  the  amount  of  mental  computation  needed  to  perform  tasks 
(particularly  for  spatial  tasks). 

9.  Dynamic  information  (i.e.,  information  that  changes  over  time)  shall  be  presented  in  real 
time  and  on  demand  to  ensure  accurate  and  timely  decision-making. 

10.  The  ESS  shall  be  flexible  enough  to  allow  for  different  operator  styles  and  responses 
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without  imposing  new  tasks  on  operators  or  affecting  overall  system  performance. 


Discussion: 

2.  Simplicity  for  the  operator  is  achieved  by  attaining  compatibility  between  the  design  and 
human  perceptual,  physical,  cognitive,  and  dynamic  motor  responsiveness  capabilities. 

3.  Objects,  actions,  and  options  shall  be  visible  to  the  operator  at  all  times.  The  operator  shall 
not  have  to  remember  information  from  one  part  of  the  dialogue  to  another.  Instructions  for 
use  of  the  system  shall  be  visible  or  easily  retrievable  whenever  appropriate. 

5.  Consistency  can  be  obtained  by  presenting  information  in  predictable  locations  and 
keeping  elements  of  screens  such  as  headers,  fields,  and  labels  consistent  in  appearance 
and  relative  location  throughout  a  system  or  application. 

6.  Navigational  aids  can  be  a  visually  or  cognitively  salient  object  whose  location  can  be 
associated  with  the  locations  of  other  objects.  Landmarks,  for  instance,  help  people  form  a 
mental  model  for  an  information  space.  Because  people  perceive  other  objects  in  relation 
to  this  point  of  reference,  a  landmark  will  organize  a  space  when  people  are  searching  for 
information  (navigation). 


7.  By  applying  human  perceptual  and  memory  characteristics  to  interface  design,  the  amount 
of  work  an  operator  must  exert  in  order  to  understand  the  information  being  presented  can 
be  reduced  and  allow  the  operator  to  focus  on  important  information. 


Evaluation  Measures  and  Methodoloaies: 

Design: 

Inspect 

Demonstrate 

Experiment 

•  Assess  overall  OMI  design  is  compliant  with 
other  relevant  Human  Factors  standards)  and 
usability  ‘heuristics’. 

0 

□ 

□ 

•  Objective  measures  of  the  adequacy  of  OMI 
ease  of  use  and  usefulness  through  Usability 
testing  (e.g.,  observation-based  studies). 

□ 

□ 

0 

•  Subjective  assessment  (i.e. ,  questionnaire)  of 
the  adequacy  of  system  usability  and  utility  by 
Human  Factors  professional  using  ‘walk¬ 
through’  or  heuristic  analysis. 

□ 

0 

□ 

Operator  Acceptance: 

•  Questionnaire-based  assessment  of  the 
adequacy  of  the  operator’s  perceptions  of 
system  usability  and  utility 

□ 

□ 

0 

Relationship  to  Other  Guidelines: 


•  3. 3. 1.1  Employ  Operator-Centered  Principles 

•  3.3.2  Employ  Operator-centered  OMI  Design 
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• 

3.5.1  Adopt  Operator-Centered  Design  Principles 

3.3.3 

Support  Different  Modes  of  Operation 

Source:  HFDS,  2003:  Nielsen,  1994: 

MS1472F;  DefStan,  1996. 

Guideline(s): 

1 .  Modes  shall  be  used  in  complex  ESSs  to  partition  the  possible  operator  actions  so  that  not 

all  tasks/actions  are  available  at  the  same  time.  Modes  shall  be  used  where  the  operator  is 
likely  to  remain  in  a  system  mode  for  a  period  of  at  least  some  minutes. 

2.  When  control,  display,  or  automation  functions  change  in  different  modes  of  ESS 
operation,  mode  and  function  identification  and  status  shall  be  clear  and  distinct  to  the 
operator  by  providing  feedback  and  clear  status  indicators  (e.g.,  sound  effects  or  visual 
indication). 

3.  Seldom-used  ESS  modes  and  functions  shall  be  clearly  identified.  As  ESSs  become  more 
complex  with  many  modes  and  functions,  the  cognitive  burden  caused  by  the  need  for 
mode  awareness  increases.  Seldom-used  ESS  modes  and  functions  will  pose  the  largest 
burden  on  the  operator  because  of  a  lack  of  familiarity.  Enabling  the  operator  to 
immediately  recognize  the  purpose  of  ESS  modes  and  functions  can  lessen  this  burden. 

4.  Frequently  used  ESS  modes  shall  be  more  accessible  than  infrequently  used  ESS  modes. 

5.  The  number  of  different  modes  for  a  given  ESS  shall  be  minimized. 

6.  The  operator  shall  be  able  to  easily  switch  between  ESS  modes. 

7.  The  ESS  shall  alert  the  operator  to  the  implications  of  interactions  between  modes, 
especially  when  they  are  potentially  hazardous. 

8.  The  ESS  shall  either  prevent  the  use  of  potentially  unsafe  modes  or  alert  the  operator  that 
a  particular  mode  may  be  hazardous. 


Discussion: 

1 .  Modes  can  be  a  frequent  source  of  operator  error  because  operators  often  mistake  the 
current  mode,  often  from  a  lack  of  effective  feedback  on  the  state  of  the  ESS  (including 
which  mode  is  active).  For  example,  a  flight  management  system  might  include  modes 
relating  to  the  cruise  and  descent  phases  of  the  flight.  In  this  case,  the  same  cockpit 
controls  are  used  to  manipulate  different  flight  variables  (e.g.,  speed  versus  descent  rate) 
according  to  which  mode  the  pilot  has  selected.  If  it  is  not  clear  to  the  pilot  which  mode  the 
automation  is  in,  potentially  dangerous  flight  parameters  could  be  inputted  inadvertently 
into  the  system. 

2.  Related  systems  shall  have  identical  coding  strategies,  identical  access  and  execution  of 
system  commands,  consistent  data  display  formatting,  and  consistent  monitoring  and 
reporting  of  resources. 

5.  Multiple  modes  will  provide  a  means  of  flexibility  but  will  introduce  more  opportunities  for 
error.  Furthermore,  a  system  that  has  multiple  modes  of  operation  can  be  difficult  to  learn 
and  can  produce  increases  in  workload. 
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Evaluation  Measures  and  Methodoloqies: 

Operator  Situation  Awareness: 

Inspect 

Demonstrate 

Experiment 

•  Subjective  assessment  (i.e.,  questionnaire)  of 
the  adequacy  of  operator’s  Situation  Awareness 
of  system  functioning  in  particular  mode  status 
(i.e.,  ‘mode  awareness’). 

□ 

□ 

0 

Operator  Workload 

•  Subjective  assessment  (i.e.,  questionnaire)  of 
the  adequacy  of  the  operator’s  workload. 

□ 

□ 

0 

Relationship  to  Other  Guidelines:  None 

3.3.4 


Provide  System  Response  and  Feedback 


Source:  HFDS,  2003;  DefStan,  1996; 
Parasuraman  and  Riley,  1999; 
MS1472F;  PISA,  1996;  Endsley,  1996. 


Guideline(s): 

1 .  The  ESS  shall  continuously  inform  the  operator  about  what  it  is  doing,  the  purpose  for 
doing  so  and  how  it  is  interpreting  the  operator’s  input.  For  every  operator  action,  there 
shall  be  some  feedback  from  the  ESS.  For  frequent  and  minor  actions,  the  response  can 
be  modest  while  for  infrequent  and  major  actions,  the  response  shall  be  more  substantial. 

2.  Feedback  messages  shall  be  phrased  in  a  clear  and  precise  manner  and  the  use  of 
abbreviations,  and  reference  system  shall  be  avoided. 

3.  The  ESS  shall  provide  a  positive  feedback  to  the  operator  regarding  the  acceptance  or 
rejection  of  a  data  entry.  When  fixed-function  key  activation  does  not  result  in  an 
immediately  observable  response  from  the  ESS,  the  operator  shall  be  given  an  indication 
of  ESS  acknowledgment. 

4.  The  ESS  shall  keep  the  operator  aware  on  a  continuing  basis  of  the  function  (or 
malfunction)  of  each  automated  sub-system  and  the  results  of  that  function  (or 
malfunction).  It  is  important  to  keep  the  operator  ‘in-the-loop’  (i.e.,  provide  sufficient 
transparency  of  the  system  for  the  operator  to  maintain  adequate  awareness  of  the 
system’s  functioning). 

5.  The  ESS  shall  alert  the  operator  when  a  problem  or  situation  is  beyond  its  capability. 

6.  The  ESS  shall  alert  the  operator  to  any  new/important  developments  occurring  in  the 
processing  and  predicting  of  outcomes  and  models. 

7.  Response  times  shall  be  as  fast  as  possible.  Normally,  no  special  feedback  is  necessary 
during  delays  of  more  than  .01  second  but  less  than  1.0  second.  For  delays  between  2 
and  10  seconds,  a  “busy”  indicator  shall  be  given  to  indicate  how  much  computer 
processing  has  been  done.  For  delays  longer  than  10  seconds,  percent-done  progress 
feedback  is  to  be  given  to  indicate  when  the  computer  expects  to  be  done  (e.g.,  percent- 
done  indicator). 
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Discussion: 


4.  When  feedback  is  poor,  ESS  functioning  is  considered  ‘silent’.  Silent  automation  may 
result  in  coordination  and  system  failures.  Operators  may  be  surprised  by  the  behaviour  of 
silent  automation. 


Evaluation  Measures  and  Methodoloqies: 

Design: 

Inspect 

Demonstrate 

Experiment 

•  Assess  compliance  with  relevant  standards  and 
usability  ‘heuristics’  concerning  system  feedback 
(e.g.,  system  response  latencies). 

0 

□ 

□ 

Operator  Situation  Awareness: 

•  Subjective  assessment  (i.e. ,  questionnaire)  of 
the  adequacy  of  the  operator’s  Situation 
Awareness  of  ESS  functioning. 

□ 

□ 

0 

Operator  Performance: 

•  Objective  assessment  of  the  adequacy  of  the 
operator’s  accurate  and  timely  detection  of  ESS 
faults  and  failures. 

□ 

□ 

0 

Relationship  to  Other  Guidelines: 


•  3.3.1 .4  Support  operator  monitoring  of  ESS  functioning 


•  3. 4. 2. 4  Maximize  Operator  Situation  Awareness  by  Increasing  System  Transparency 


i _ i 

3.3.5 

Support  Identification  and  Management  of 

ESS  Faults  and  Failures 

Source:  HFDS,  2003:  DefStan,  1996: 
MS1472F;  Parasuraman  and  Riley, 

1997. 

Guideline(s): 


1 .  Make  ESS  failures  apparent  by  making  failure  unambiguously  obvious  to  the  operator 

2.  Provide  adequate  early  warning  notification  of  pending  ESS  failure  or  performance 
decrements  to  allow  the  operator  to  adjust  to  the  new  task  load  and  take  manual  control. 

3.  Inform  the  operator  of  potential  ESS  failure  and  malfunctions. 

4.  The  first  alarm  event  shall  be  clearly  identifiable  so  that  the  operator  is  able  to  identify  the 
first  event  in  case  of  a  series  of  alarm  events. 

5.  Provide  sufficient  diagnostic  information  that  is  self-explanatory  and  in  plain  English. 

6.  Error-prone  conditions  shall  be  minimized  by  maintaining  operator  awareness,  providing 
adequate  training  and  developing  standard  operating  procedures. 
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Discussion: 

1 .  Stress,  preoccupation,  and  distraction  may  reduce  the  operator’s  ability  to  detect  faults. 

2.  In  situations  where  ESS  failure  would  require  operator  intervention,  it  is  useful  for  the 
operator  to  be  warned  that  he  or  she  will  need  to  take  manual  control  before  the  ESS  fails. 
Ideally,  this  warning  needs  to  come  in  adequate  time  to  allow  the  operator  to  adjust  to  the 
new  task  load.  There  may,  however,  be  cases  where  it  is  not  possible  to  provide  advance 
notification  of  pending  failure  or  where  the  estimate  of  time  needed  for  the  operator  to  take 
control  is  unknown. 

3.  It  can  increase  workload  for  the  operator  to  continually  monitor  the  ESS  for  failure. 
Advance  knowledge  about  potential  failures  can  also  help  the  operator  prepare  to  take 
manual  control. 

5.  In  order  for  the  operator  to  diagnose  the  ESS,  diagnostics  information  needs  to  be  self- 
explanatory  and  in  plain  language.  The  diagnostic  information  must  provide  the  operator 
with  the  information  they  need  without  requiring  the  operator  to  seek  additional  references, 
or  a  help  function,  to  understand  the  problem  and  the  recommended  solution. 

6.  Errors  may  arise  from  data  entry  errors,  monitoring  failures,  system  workarounds,  and 
mode  misapplication.  Error-prone  conditions  in  ESSs  may  result  from  lack  of  mode 
awareness,  lack  of  situation  awareness,  lack  of  systems  awareness,  increased  heads 
down  time,  over-dependence  on  automation,  and  interrupted  crew  coordination. 
Automation-related  errors  usually  occur  in  conjunction  with  other  factors  such  as  haste, 
inattention,  fatigue,  or  distraction. 


Evaluation  Measures  and  Methodoloqies: 

Design: 

Inspect 

Demonstrate 

Experiment 

•  Assess  compliance  with  relevant  standards  and 
usability  ‘heuristics’  concerning  the  design  of 
alarms  and  warnings. 

0 

□ 

□ 

Operator  Situation  Awareness: 

•  Subjective  assessment  (i.e. ,  questionnaire)  of 
the  adequacy  of  the  operator’s  Situation 
Awareness  of  ESS  functioning. 

□ 

□ 

0 

•  Objective  assessment  (e.g.,  probe  technique)  of 
the  adequacy  of  the  operator’s  ability  to 
anticipate  future  ESS  failures. 

□ 

□ 

0 

Operator  Workload: 

•  Subjective  assessment  (i.e.,  questionnaire)  of 
the  adequacy  of  the  operator’s  workload. 

□ 

□ 

0 
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Operator  Trust: 

•  Subjective  assessment  (i.e.,  questionnaire)  of 
the  adequacy  of  the  operator’s  trust  in  the  ESS 
functioning. 

□ 

□ 

0 

Operator  Performance: 

•  Objective  assessment  of  the  adequacy  of  the 
operator’s  accurate  and  timely  detection  of  ESS 
faults  and  failures. 

□ 

□ 

0 

•  Objective  assessment  of  the  adequacy  of  the 
operator’s  accurate  and  timely  management  of 
ESS  faults  and  failures. 

□ 

□ 

0 

•  Objective  assessment  of  the  adequacy  of  the 
operator’s  ability  (e.g.,  speed  and  error)  to 
resume  manual  control. 

□ 

□ 

0 

Relationship  to  Other  Guidelines: 


•  3.3.1 .3  Promote  ESS  Robustness  and  Resilience  to  Operator  Error 


3.4  Class-Specific  Guidelines 


3.4.1  Information  Management  Aids 


3.4. 1.1 

Optimize  Information  Presentation  and 

Source:  HFDS,  2003:  DISA,  1996: 

Information  Management 

DefStan,  1996;  Hutchins  et  al.,  1999; 
AHCI,  1998. 

Guideline(s): 


1.  Information  presented  to  the  operator  shall  accurately  reflect  system  and  environment 
status  in  a  manner  so  that  the  operator  rapidly  recognizes,  easily  understands,  and  easily 
projects  system  outcomes  in  relation  to  system  and  operator  goals. 

2.  Data  changes  that  occur  following  Information  Management  Aid  display  update  shall  be 
temporarily  highlighted. 

3.  Reduce  amount  of  information  the  operator  must  evaluate. 

4.  The  Information  Management  Aid  shall  provide  both  information  about  an  object’s  features 
and  explanatory  descriptions  which  support  various  decision  making  processes.  For 
example,  a  track’s  features  determining  its  intent  and  capability  (i.e.,  its  “threatiness”)  shall 
be  displayed  and/or  easily  accessible. 

5.  The  Information  Management  Aid  shall  be  able  to  effectively  evaluate,  integrate  and 
present  information  to  the  operator  so  that  an  accurate  synthesized  picture  of  the  situation 
is  achieved. 
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6.  Information  presented  by  the  Information  Management  Aid  shall  be  clear,  meaningful, 
consistent,  legible,  discriminable,  and  structured  and  based  on  an  understanding  of  the 
tasks  performed  by  operators. 

7.  Information  shall  be  unambiguous  and  meaningful  to  the  operator. 

8.  When  information  must  be  updated  quickly,  the  most  important  information  shall  be  cued 
to  ensure  it  will  be  the  first  to  be  processed  by  the  operator.  It  is  important  that  the  cues 
shall  be  correct,  as  there  may  be  significant  costs  of  invalid  cueing. 

9.  Incoming  messages  shall  be  queued  automatically  by  the  Information  Management  Aid  so 
they  do  not  disrupt  current  information  handling  tasks. 

10.  Long  lists  of  information,  tasks,  and  so  on,  shall  be  stored  and  prioritized  by  the  ESS  to 
minimize  the  number  of  decision  alternatives  and  reduce  the  visual  processing  load  of 
human  operators. 

11.  Information  Management  Aid  information  shall  be  automatically  reorganized  into  integrated 
or  non-integrated  arrangements  depending  on  the  current  system  status. 

12.  The  Information  Management  Aid  shall  provide  accurate  and  reliable  information.  That  is, 
the  correctness  of  the  information  in  reflecting  the  reality. 

1 3.  The  Information  Management  Aid  shall  automatically  notify  the  operator  of  meaningful 
patterns  or  events  such  as  when  it  predicts  a  future  problem. 

14.  The  Information  Management  Aid  shall  present  information  at  the  level  of  detail  that  is 
appropriate  to  the  immediate  task,  with  no  more  information  than  is  essential. 

15.  The  Information  Management  Aid  shall  avoid  repeating  information  that  is  already 
available. 

1 6.  The  Information  Management  Aid  shall  be  fully  integrated  and  consistent  with  the  rest  of 
the  OMI. 


Discussion: 

1.  Communication  will  be  improved  by  allowing  information  to  be  presented  in  the  most 
understandable  format.  Eliminating  the  need  to  translate  information  into  a  specific  format 
will  reduce  workload. 

2.  A  primary  objective  of  information  automation  is  to  maintain  and  enhance  situation 
awareness.  However,  too  much  information  presented  simultaneously  may  become 
cluttered  and  make  visual  search  difficult,  interfering  with  status,  decision-making,  or 
control.  It  is  important  for  the  operator  to  be  able  to  easily  locate  needed  information.  The 
operator’s  ability  to  detect  a  signal  while  monitoring  varies  inversely  with  the  rate  at  which 
neutral  background  events  are  repeated.  There  is  also  good  evidence  that  the  ability  to 
accurately  define  an  event  as  a  signal  is  improved  if  the  operator  has  a  good 
understanding  of  what  a  non-signal  is. 

7.  Where  information  coding  techniques  are  used,  the  meaning  associated  with  codes  shall 
_ be,  as  far  as  possible,  based  on  associations  with  which  the  target  population  can  be 
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expected  to  be  familiar  (such  as  "Red  =  Danger").  Words,  names  and  abbreviations  shall 
be  based  on  language  and  terminology  used  by  the  target  operator  population.  Data 
parameters  and  units  shall  use  formats  which  are  meaningful  to  the  target  operators  and 
consistent  with  the  overall  task  context. 

11.  Integrated  information  arrangement  allows  the  operator  to  assess  the  overall  status  of  the 
system.  Integrating  display  components  into  aggregated  arrangements  may  reduce  the 
attention  demands  of  fault  detection.  Non-integrated  arrangement  of  components  draws 
operator  attention  to  system  errors  or  other  relevant  information  (i.e. ,  ‘pop-out’). 


12.  Accurate  and  reliable  information  will  increase  the  operator’s  trust  in  the  system,  support 
the  decision  making  process  and  increase  the  likelihood  of  an  appropriate  course  of 
action. 


Evaluation  Measures  and  Methodoloqies: 

Design: 

Inspect 

Demonstrate 

Experiment 

•  Assess  compliance  with  relevant  standards  and 
usability  ‘heuristics’  (e.g.,  Nielsen,  1994) 
concerning  the  presentation  of  information. 

0 

□ 

□ 

System  Performance: 

•  Percentage  of  data  objects  correctly  identified 
and  prioritized. 

□ 

0 

□ 

•  Percentage  of  misses  and  false  positives. 

□ 

0 

□ 

•  Age  of  information. 

Operator  Performance: 

□ 

0 

□ 

•  Objective  assessment  of  the  adequacy  of  the 
operator’s  accurate  and  timely  detection  and 
management  of  key  events 

□ 

□ 

0 

Operator  Situation  Awareness: 

•  Subjective  assessment  (i.e.,  questionnaire)  of 
the  adequacy  of  the  operator’s  Situation 
Awareness  of  task-relevant  objects  in  the 
environment 

□ 

□ 

0 

Operator  Workload: 

•  Subjective  assessment  (i.e.,  questionnaire)  of 
the  adequacy  of  the  operator’s  workload 

□ 

□ 

0 

Relationship  to  Other  Guidelines: 


•  3. 3. 1.1  Employ  Operator-Centered  Guidelines 
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3.4.1 


.2 


Optimize  Display  of  Information 


Source:  HFDS,  2003;  Hutchins  et  al., 
1999;  DefStan,  1996;  Zachary  and 
Ryder,  1997. _ 


Guideline(s): 

1.  Event  data  shall  be  combined  with  a  map  background  when  the  geographic  location  of 
changing  events  needs  to  be  shown.  This  might  be  implemented  as  an  operator-selectable 
function  to  avoid  unnecessary  levels  of  clutter. 

2.  Integrated  displays  shall  combine  various  Information  Management  Aid  information 
elements  into  a  single  representation. 

3.  Dynamic  data  that  must  be  monitored  by  the  operator  shall  be  displayed  as  a  graphic 
format. 


4.  Automated  (i.e.,  ESS  instigated)  and  non-automated  (i.e.,  operator  instigated)  cues  shall 
be  made  equally  prominent  to  enable  operators  to  collect  confirming/disconfirming 
evidence  before  deciding  on  appropriate  action. 

5.  Provide  operators  with  displays  (e.g.,  representional  aids)  that  allow  them  to  see  directly 
the  information  they  require  rather  than  infer  it  from  using  more  cognitively  intense  levels  of 
data  processing. 

6.  Display  elements  shall  only  be  integrated  if  it  will  enhance  status  interpretation,  decision¬ 
making,  situation  awareness,  or  other  aspects  of  task  performance. 

7.  Visual  representations  of  data  shall  be  used  to  (1 )  present  huge  amounts  of  data;  (2)  show 
emergent  properties  of  large  amounts  of  data  and  (3)  separate  multiple  dimensions  within 
a  single  representation. 

8.  Graphical  displays  of  information  shall  be  used  to  reduce  the  amount  of  mental  processing 
by  allowing  operators  to  spend  less  time  searching  for  information. 

9.  Visual  representations  of  information  shall  be  used  to  represent  data  relationships. 

10.  Provide  meaningful  patterns  of  information  by  matching  the  operator’s  expertise  in  the 
domain  (skills  and  knowledge  of  the  domain). 


Discussion: 

5.  Integrated  information  arrangement  allows  the  operator  to  assess  the  overall  status  of  the 
system.  Integrating  display  components  into  aggregated  arrangements  may  reduce  the 
attention  demands  of  fault  detection.  Non-integrated  arrangement  of  components  draws 
operator  attention  to  system  errors  or  other  relevant  information  (i.e.,  ‘pop-out’).  Presenting 
the  information  in  a  format  relevant  to  the  state  of  the  system  can  facilitate  the  ability  of  the 
operator  to  quickly  and  easily  assess  the  system  status. 

7.  A  large  amount  of  data  (e.g.,  parametric)  could  be  portrayed  graphically  for  rapid 
assimilation  by  the  operator.  For  instance,  the  operator  could  see,  at  a  glance,  a 
synthesized  picture  of  the  track’s  behaviour.  Compared  with  more  complex  logical 
operations,  this  rather  simple  perceptual  operation  allows  operators  to  omit  steps  that  are 
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otherwise  necessary  when  a  task  is  performed  without  a  visual  representation. 

8.  The  graphical  presentation  of  information  allows  operators  to  substitute  less  demanding 
perceptual  operations  for  more  complex  logical  operations.  That  is,  graphical  displays 
allow  decision  makers  to  “see”  directly  the  information  they  require  rather  than  infer  it.  For 
example,  determining  a  change  in  altitude  (and  the  degree  of  change)  can  be  immediately 
apparent  when  the  operator  glances  at  a  line  graph  depicting  a  track’s  history.  Meanwhile, 
graphics  can  help  operators  save  time  when  searching  for  needed  information  when 
several  related  dimensions  of  information  are  encoded  in  a  single  graphical  object.  Novel 
graphical  displays  must  be  evaluated  carefully  to  ensure  that  the  operator  interprets  the 
graphical  information  in  the  way  intended  by  the  designer. 


9.  The  Information  Management  Aid  can  visually  represent  (1)  system  relationships,  its  rule 
network,  and  reasoning  process;  (2)  visual  associations  between  related  information;  and 
(3)  new  relationships  previously  seen  as  unrelated. 


Evaluation  Measures  and  Methodoloqies: 

Design: 

Inspect 

Demonstrate 

Experiment 

•  Assess  compliance  with  relevant  standards,  and 
usability  ‘heuristics’  concerning  the  presentation 
of  information. 

0 

□ 

□ 

Operator  Situation  Awareness: 

•  Subjective  assessment  (i.e. ,  questionnaire)  of 
the  adequacy  of  the  operator’s  Situation 
Awareness  of  task-relevant  objects  in  the 
environment. 

□ 

□ 

0 

Operator  Workload: 

•  Subjective  assessment  (i.e.,  questionnaire)  of 
the  adequacy  of  the  operator’s  workload. 

□ 

□ 

0 

Relationship  to  Other  Guidelines:  None 


3.4.2 

Decision  Making  Aids 

3.4.2. 1 

Ensure  Appropriate  Implementation 

Source:  HFDS,  2003:  Hutchins  et  al., 

1 999;  ACHI,  1 998;  Zachary  and 

Ryder,  1997;  DISA,  1996; 

Parasuraman  and  Riley,  1997. 

Guideline(s): 

1. 

Decision  Making  Aids  shall  be  used:  for  managing  system  complexity;  for  assisting 
operators  in  coping  with  information  overload;  for  focusing  the  operator’s  attention;  for 
assisting  the  operator  in  accomplishing  time-consuming  activities  more  quickly;  when 
limited  data  results  in  uncertainty;  for  overcoming  human  limitations  that  are  associated 
with  uncertainty,  the  emotional  components  of  decision-making,  finite-memory  capacity, 
and  systematic  and  cognitive  biases;  and,  for  assisting  the  operator  in  allocating 

44 


resources,  managing  detailed  information,  performing  computations,  and  selecting  and 
deciding  among  alternatives. 

2.  The  Decision  Making  Aid  shall  not  be  implemented  when  solutions  are  obvious;  when  one 
alternative  clearly  dominates  all  other  options;  when  there  is  insufficient  time  to  act  upon  a 
decision;  when  the  operator  is  not  authorized  to  make  decisions;  or  for  cognitive  tasks  in 
which  humans  excel,  including  generalization  and  adapting  to  novel  situations. 

3.  The  Decision  Making  Aid  shall  assist,  rather  than  replace,  human  decision  makers  by 
providing  data  for  making  judgments  rather  than  commands  that  the  operator  must 
execute. 

4.  The  operator  shall  be  able  to  configure  the  Decision  Making  Aid  to  provide  the  kind  and 
level  of  support  he/she  requires. 


Discussion: 

1 .  The  objective  of  a  Decision  Making  Aid  is  to  increase  the  speed  of  analysis  of  tactical  data; 
allow  for  more  accurate  course  of  action  and  decision  timeliness  and  agility. 

3.  Research  has  shown  that  experienced  decision  makers  recognize  the  situation  or  scenario 
based  on  comparison  of  the  features  of  the  current  situation  with  stored  memory 
representations.  Once  the  situation  is  recognized,  solutions  or  course  of  action  are 
stimulated  by  activation  of  these  memory  representations. 


4.  Operators  shall  be  able  to  determine  when  and  how  the  Decision  Making  Aid  should  be 
used. 


Evaluation  Measures  and  Methodoloqies: 

Operator  Performance  (Decision  Making  Quality): 

Inspect 

Demonstrate 

Experiment 

• 

Subjective  assessment  (i.e. ,  FASA 
questionnaire)  of  the  adequacy  of  the  quality  of 
the  operator’s  situation  assessment  processes. 

□ 

□ 

0 

• 

Objective  assessment  of  the  adequacy  of  the 
timeliness  and  agility  of  operator  decision 
making. 

□ 

□ 

0 

• 

Objective  assessment  of  the  adequacy  of  the 
consistency  of  operator  decision  making. 

□ 

□ 

0 

• 

Subjective  assessment  (e.g.,  peer  review)  of  the 
adequacy  of  the  justifiability  and  rationality  of 
operator  decision  making. 

□ 

□ 

0 

Operator  Trust: 

• 

Subjective  assessment  (i.e.,  questionnaire)  of 
the  adequacy  of  Operator  trust.  Excessive  levels 
of  trust  may  indicate  operator  complacency 
(over-trust)  and  very  low  levels  of  trust  may 

□ 

□ 

0 
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indicate  operator  scepticism. 

Operator  Workload: 

•  Subjective  assessment  (i.e. ,  questionnaire)  of 
the  adequacy  of  the  operator’s  workload. 

□ 

□ 

0 

Relationship  to  Other  Guidelines: 

•  3. 4. 2. 2  Support  decision  making  strategies 

3.4.2 


.2 


Support  decision  making  strategies 


Source:  HFDS,  2003;  Hutchins  et  al., 
1999;  ACHI,  1 998;  Zachary  and 
Ryder,  1997;  DISA,  1996; 
Parasuraman  and  Riley,  1997. _ 


Guideline(s): 

1.  The  Decision  Making  Aid  shall  support  decision  alternatives. 

2.  When  more  than  one  alternative  is  available,  the  Decision  Making  Aid  shall  provide 
alternatives  in  a  recommended  prioritization  scheme  based  on  mission  and  task  analysis. 

3.  When  the  information  used  by  a  Decision  Making  Aid  is  derived  or  processed,  the  data 
from  which  it  is  derived  shall  be  either  visible  or  accessible  for  verification. 


4.  The  Decision  Making  Aid  shall  be  capable  of  planning  a  strategy  to  address  a  problem  or 
guide  a  complex  process. 

5.  Develop  models  of  decision  making  strategies  specific  to  the  domain  and  the  mission. 
From  the  decision  making  model,  the  type  of  information  to  display  and  how  to  display  it 
will  become  evident. 


6.  The  Decision  Making  Aid  shall  keep  the  number  of  response  options  to  a  manageable 
number. 


7.  The  support  provided  by  the  Decision  Making  Aid  shall  be  consistent  with  operator 
cognitive  strategies  and  expectations  (mental  models).  A  mental  model  is  an  individual’s 
understanding  of  the  processes  underlying  system  operation. 

8.  The  Decision  Making  Aid  shall  be  able  to  predict  future  data  based  on  historical  data  and 
current  conditions. 


9.  The  Decision  Making  Aid  shall  minimize  queries  by  the  operators  for  information. 

1 0.  The  Decision  Making  Aid  shall  be  tailored  to  the  expertise  and  skill  of  the  decision  maker 
and  the  support  for  one  level  of  expertise  shall  not  interfere  with  the  support  for  operators 
with  different  levels  of  expertise. 


Discussion: 

1.  Arguments  leading  to  system  results  and  alternative  solutions  shall  be  displayed  so  that 
the  operator  is  able  to  comprehend  and  evaluate  computer-generated  proposals  and 
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formulate  a  well-informed  decision. 

3.  Data  that  are  not  critical  for  an  operation  can  be  made  available  only  upon  request. 

4.  Ensure  that  the  Decision  Making  Aid  guides  the  operator  through  the  process,  providing 
automated  guidance  on  how  to  define  and  analyze  a  problem  and  formulate  a  decision. 

6.  The  number  of  options  that  the  operator  must  consider  is  expected  to  decrease  when  a 
Decision  Making  Aid  is  used.  Reducing  the  response  options  focuses  the  operator’s 
attention  onto  the  most  viable  options.  However,  presenting  too  few  options  might  promote 
‘cognitive  tunnel  vision’,  which  should  be  avoided. 

10.  For  instance,  novices  to  the  domain  may  rely  on  a  rule-based  interface  (i.e.,  analytical)  to 
aid  in  their  decision  process  while  experts  may  be  more  responsive  to  mental  imaging 
techniques  (e.g.,  Recognition-Primed  Decision  or  feature  matching).  Novices  typically  have 
good  knowledge  of  the  domain  (i.e.,  C2  processes)  but  it  is  composed  largely  of  facts  and 
basis  concepts.  Decision  making  strategies  generally  involve  analyzing  individual  multiple 
variables  and  applying  general,  domain-independent  problem  solving  methods  (e.g.,  trial 
and  error).  As  the  operator  becomes  more  adept,  the  consolidated  knowledge  and 
problem  solving  approaches  support  the  use  of  seemingly  more  recognition/case-based 
approaches.  Such  an  operator  has  developed  a  complex  set  of  integrated  declarative  (the 
“what”)  and  procedural  (the  “how  to”)  knowledge  both  about  the  domain  and  about  the 
decision  tasks  to  be  performed  in  the  domain.  This  rich  mental  model  of  the  domain  allows 
the  operator  to  apply  elaborated  domain-based  solutions.  The  Decision  Making  Aid  shall 
be  designed  to  support  the  expert  decision  maker  to  recognize  the  case  into  which  a 
specific  decision  situation  fits.  The  expert  decision  maker  can  then  retrieve  the  appropriate 
solution  strategy  and  directly  apply  it  with  little  or  no  intermediate  analysis  or  reasoning. 


Evaluation  Measures  and  Methodoloqies: 

Operator  Decision  Making  Quality: 

Inspect 

Demonstrate 

Experiment 

• 

Subjective  assessment  (i.e.,  FASA 
questionnaire)  of  the  adequacy  of  the  quality  of 
the  operator’s  situation  assessment  processes. 

□ 

□ 

0 

• 

Objective  assessment  of  the  adequacy  of  the 
timeliness  and  agility  of  operator  decision 
making. 

□ 

□ 

0 

• 

Objective  assessment  of  the  adequacy  of  the 
consistency  of  operator  decision  making. 

□ 

□ 

0 

• 

Subjective  assessment  (e.g.,  peer  review)  of  the 
adequacy  of  the  justifiability  and  rationality  of 
operator  decision  making. 

□ 

□ 

0 

Operator  Situation  Awareness: 

• 

Subjective  assessment  (i.e.,  questionnaire)  of 
the  adequacy  of  the  operator’s  Situation 
Awareness  of  the  reasoning  behind 
recommendations  from  the  Decision  Making  Aid 

□ 

□ 

0 

47 


(i.e.,  system  transparency). 

Operator  Workload: 

•  Subjective  assessment  (i.e.,  questionnaire)  of 
the  adequacy  of  the  operator’s  workload. 

Operator  Trust: 

•  Subjective  assessment  (i.e.,  questionnaire)  of 
the  adequacy  of  operator  trust.  Excessive  levels 
of  trust  may  indicate  operator  complacency 
(over-trust)  and  very  low  levels  of  trust  may 
indicate  operator  scepticism. 


□ 


□ 


□ 


□ 


0 


0 


_ L 

Relationship  to  Other  Guidelines: 

•  3.4.2. 1  Ensure  Appropriate  Implementation 

_ i _ i _ 

3. 4.2. 3 

Keep  Operators  in  Control 

Source:  HFDS,  2003:  DefStan,  1996: 
MS1472F;  Parasuraman  and  Riley, 
1997;  Sheridan,  2000. 

Guideline(s): 

1.  Operators  shall  be  able  to  determine  when  and  how  the  Decision  Making  Aid  shall  be 
used. 

2.  The  Decision  Making  Aid  shall  not  be  able  to  veto  operator  actions  leaving  the  operator 
without  means  to  override  or  violate  rules  that  govern  the  Decision  Making  Aid  unless 
there  is  not  enough  time  for  the  operator  to  make  a  decision. 

3.  The  operator  shall  be  able  to  initiate  (i.e.,  over-ride)  the  automation  of  tasks  even  when  a 
task  has  been  designated  to  be  Decision  Making  Aid-initiated. 

4.  The  Decision  Making  Aid  shall  assist,  rather  than  replace,  human  decision  makers  by 
providing  data  for  making  judgments  rather  than  commands  that  the  operator  must 
execute. 

5.  The  Decision  Making  Aid  shall  allow  the  operator  to  receive  direct  assistance  in  planning 
how  to  carry  out  the  intended  task. 

6.  The  Decision  Making  Aid  shall  accept  direction  from  the  operators  on  which  problem 
solving  strategy  to  employ  when  alternative  strategies  are  available. 

7.  Automated  tasks  or  functions  shall  not  be  able  to  jeopardize  safety  or  make  a  difficult 
situation  worse. 

8.  When  an  operator  might  need  to  operate  in  out-of-tolerance  conditions,  then  a  deliberate 
overriding  action  shall  be  possible. 
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Discussion: 

1 .  Operator  acceptance  of  a  Decision  Making  Aid  centres  on  whether  the  operator  feels  in 
control  of  the  system. 


8.  There  may  be  cases,  particularly  in  an  emergency  situation,  when  the  operator  needs  to 
operate  in  out-of-tolerance  conditions. 


Evaluation  Measures  and  Methodoloqies: 

Operator  Trust 

Inspect 

Demonstrate 

Experiment 

•  Subjective  assessment  (i.e. ,  questionnaire)  of 
the  adequacy  of  operator  trust.  Excessive  levels 
of  trust  may  indicate  operator  complacency 
(over-trust)  and  very  low  levels  of  trust  may 
indicate  operator  scepticism. 

□ 

□ 

0 

Operator  Situation  Awareness 

•  Subjective  assessment  (i.e.,  questionnaire)  of 
the  adequacy  of  the  operator’s  Situation 
Awareness  of  task-relevant  objects  in  the 
environment. 

□ 

□ 

0 

Operator  Workload: 

•  Subjective  assessment  (i.e.,  questionnaire)  of 
the  adequacy  of  the  operator’s  workload. 

□ 

□ 

0 

Relationship  to  Other  Guidelines: 


•  3.3.5  Support  Identification  and  Management  of  ESS  Faults  and  Failures 


•  3.4.3. 1  Keep  Operators ‘In-the-Loop’ 


i _ i 

3. 4.2.4 

Maximize  Operator  Situation  Awareness  by 
Increasing  System  Transparency 

Source:  HFDS,  2003:  Zachary  and 
Ryder,  1997;  Hutchins  et  al. ,  1997; 
DefStan,  1996;  Endsley,  1996. 

Guideline(s): 


1.  Processed  data  shall  be  accessible. 

2.  Promote  knowledge  about  the  intent  of  the  Decision  Making  Aid  to  the  Operator. 

3.  The  Decision  Making  Aid  shall  estimate  and  indicate  the  certainty  of  analysis  and  provide 
the  rationale  for  the  estimate. 

4.  The  Decision  Making  Aid  shall  give  the  operator  access  to  procedural  information  used  by 
the  aid. 

5.  When  the  Decision  Making  Aid  provides  explanations  to  the  operator,  it  shall  supply  a 
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short  explanation  initially,  with  the  ability  to  make  available  more  detail  at  the  operator’s 
request,  including  access  to  process  information  or  an  explanation  for  the  rules, 
knowledge-basis,  and  solutions  used  by  the  decision  aid. 

6.  When  the  Decision  Making  Aid  provides  explanations  to  the  operator,  the  explanation  shall 
use  terms  familiar  to  the  operator  and  maintain  consistency  with  the  immediate  task. 

7.  The  Decision  Making  Aid  shall  alert  the  operator  to  changes  in  the  status  of  important 
system  information  such  as  when  critical  information  becomes  available  during  decision 
aid  utilization. 


Discussion: 

1.  Where  displays  contain  potentially  large  amounts  of  information,  consideration  shall  be 
given  to  providing  operators  with  facilities  to  manage  the  amount  and  types  of  information 
displayed  at  any  one  time.  This  can  be  achieved  by  applying  filters  and  artificial 
intelligence  (e.g.,  algorithms)  based  on  the  operator  role  to  help  process  the  data. 

2.  Monitoring  of  the  Decision  Making  Aid  by  the  operator  and  the  operator  by  the  system  can 
only  be  effective  if  each  knows  what  the  other  one  is  trying  to  accomplish  (i.e.,  intent).  This 
might  be  achieved  by  displaying  the  current  goals  of  the  Decision  Making  Operator  (as  well 
as  progress  made  towards  those  goals). 

3.  Research  pertaining  to  the  representation  of  system  certainty  (or  uncertainty)  to  the 
operator  is  immature.  Any  attempt  to  represent  system  certainty  (or  uncertainty)  to  the 
operator  must  be  thoroughly  evaluated. 

4.  Procedural  information  is  information  about  the  rules  or  algorithms  used  by  the  Decision 
Making  Aid.  Knowledge  of  procedural  information  fosters  operator  acceptance  of  the  aid 
because  the  operator  is  able  to  understand  how  the  aid  functions.  As  the  operator 
becomes  more  familiar  with  a  given  situation,  he  or  she  requires  less  procedural 
information. 

5.  Process  information  is  the  information  about  how  the  Decision  Making  Aid  accomplishes  a 
task.  This  information  is  required  by  operators  to  decide  whether  to  use  the  aid  in 
unfamiliar  situations  and  for  identifying  the  nature  and  extent  of  malfunctions. 


7.  Critical  information  in  this  standard  refers  to  information  that  may  have  a  significant  impact 
on  task  completion. 


Evaluation  Measures  and  Methodoloaies: 

Inspect 

Demonstrate 

Experiment 

Operator  Situation  Awareness: 

•  Subjective  assessment  (i.e.,  FASA 

questionnaire)  of  the  adequacy  of  the  quality  of 
the  operator’s  situation  assessment  processes 

□ 

□ 

0 

•  Subjective  assessment  (i.e.,  questionnaire)  of 
the  adequacy  of  the  operator’s  Situation 
Awareness  of  task-relevant  objects  in  the 
environment 

□ 

□ 

0 
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•  Subjective  assessment  (i.e. ,  questionnaire)  of 
the  adequacy  of  the  operator’s  Situation 
Awareness  of  the  reasoning  behind 
recommendations  from  the  Decision  Making  Aid 
(i.e.,  system  transparency). 

□ 

□ 

0 

Operator  Workload: 

•  Subjective  assessment  (i.e.,  questionnaire)  of 
the  adequacy  of  the  operator’s  workload. 

□ 

□ 

0 

Operator  Trust 

•  Subjective  assessment  (i.e.,  questionnaire)  of 
the  adequacy  of  operator  trust.  Excessive  levels 
of  trust  may  indicate  operator  complacency 
(over-trust)  and  very  low  levels  of  trust  may 
indicate  operator  scepticism. 

□ 

□ 

0 

Relationship  to  Other  Guidelines: 


•  3.3.1 .4  Support  Operator  Monitoring  of  ESS  Functioning 

•  3.3.4  Provide  System  Response  and  Feedback 


3.4.3  Control  and  Action  Aids 


3.4.3. 1 


Keep  Operators  ‘In-the-Loop’ 


Source:  HFDS,  2003;  Sheridan,  2000; 
MS1472F;  DefStan,  1996;  Endsley  and 
Kaber,  1999. _ 


Guideline(s): 

1 .  When  tasks  are  automated,  the  tasks  shall  be  easily  understood  by  operators  and 
matched  to  the  operator’s  mental  model  of  the  task. 

2.  The  Control  and  Action  Aid  shall  provide  the  operator  with  an  appropriate  range  of  control 
options  that  are  flexible  enough  to  accommodate  the  full  range  of  operating  conditions  for 
which  it  was  certified. 


3.  To  promote  sufficient  levels  of  operator  situation  awareness  of  the  Control  and  Action  Aid, 
the  operator  shall  be  given  immediate  feedback  to  command  and  control  orders. 

4.  Override  and  backup  control  alternatives  shall  be  available  for  automated  tasks  that  are 
critical  to  the  integrity  of  the  system  or  when  lives  depend  on  the  system. 

5.  The  operator  shall  be  able  to  initiate  and  control  the  direction  and  pace  of  the  tasks  and/or 
functions  of  the  Control  and  Action  Aid  until  the  point  at  which  operator  goals  have  been 
met, 

6.  Information  for  backup  or  override  capability  shall  be  readily  accessible. 

7.  The  Control  and  Action  Aid  shall  be  designed  so  that  operators  are  involved  in  active 
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control  and  monitoring  rather  than  just  passive  monitors. 

8.  Allow  reversal  of  operator  actions  (e.g.  ‘undo’  or  ‘cancel’  function)  and  give  clear 
indications  how  reversal  can  be  achieved. 


Discussion: 

2.  Highly  flexible  Control  and  Action  Aids  can  be  useful  when  the  operator  knows  how  to 
implement  the  various  options  across  a  wide  spectrum  of  operational  situations.  However, 
the  multiple  options  that  are  associated  with  highly  flexible  systems  also  require  additional 
cognitive  resources  in  order  for  the  operator  to  remember  which  mode  is  active. 

7.  An  active  role  will  decrease  the  likelihood  of  complacency  and  lower  vigilance  and  may 
increase  situation  awareness. 

8.  In  order  to  facilitate  the  operator’s  perception  of  being  in  control  (as  opposed  to  the 
perception  of  the  Control  and  Action  Aid  being  in  control  of  the  operator),  the  Control  and 
Action  Aid  shall  allow  the  operator  an  easy  exit  out  of  as  many  interactions  as  possible. 
For  example,  by  providing  a  cancel  button,  and  undo  and  redo  operations. 


Evaluation  Measures  and  Methodoloqies: 

Operator  Situation  Awareness: 

Inspect 

Demonstrate 

Experiment 

•  Subjective  assessment  (i.e. ,  questionnaire)  of 
the  adequacy  of  the  operator’s  Situation 
Awareness  of  the  reasoning  behind  the  actions 
of  the  Control  and  Action  Aid  (i.e.,  system 
transparency). 

□ 

□ 

0 

Operator  Trust: 

•  Subjective  assessment  (i.e.,  questionnaire)  of 
the  adequacy  of  operator  trust.  Excessive  levels 
of  trust  may  indicate  operator  complacency 
(over-trust)  and  very  low  levels  of  trust  may 
indicate  operator  scepticism. 

□ 

□ 

0 

Operator  Workload: 

•  Subjective  assessment  (i.e.,  questionnaire)  of 
the  adequacy  of  the  operator’s  workload. 

□ 

□ 

0 

Relationship  to  Other  Guidelines: 


•  3.3.1 .4  Support  Operator  Monitoring  of  ESS  Functioning 

•  3.3.4  Provide  System  Response  and  Feedback 

•  3. 4. 2. 4  Maximize  Operator  Situation  Awareness  by  Increasing  System  Transparency 
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3.5  Design  and  Evaluation 


3.5.1 


Adopt  Operator-Centered  Design  Principles 


Source:  HFDS,  2003;  DefStan,  1996; 
AHCI,  1998;  DISA,  1996;  Zachary  and 
Ryder,  1997;  MS1472F. _ 


Guideline(s): 

1 .  The  design  of  an  ESS  shall  begin  by  choosing  the  human-centered  criteria  (goals)  of  the 
system  and  then  defining  the  functions  that  the  system  will  perform. 

2.  Standard  operating  procedures  and  company  policies  shall  guide  system  designers  in  the 
appropriate  allocation  of  a  task  to  the  operator  or  the  ESS,  although  the  operator  shall  be 
ultimately  responsible  to  make  the  decision  to  use  or  not  use  the  automation. 

3.  A  substantial  proportion  of  the  operators  shall  be  involved  in  the  design  of  the  ESS. 

4.  The  ESS  shall  be  based  on  operator  population  characteristics  (cognitive  and  heuristic 
biases,  skills,  experience,  training)  and  cognitive  processes  (mental  representation  and 
decision  strategy)  of  the  operator. 

5.  The  unique  contextual  and  environmental  considerations  shall  be  incorporated  into  the 
design  of  the  ESS  to  support  decision  making. 

6.  When  a  new  ESS  technology  is  introduced,  the  designers  shall  consider  the  possibility  of 
negative  effects  on  team  coordination. 

7.  The  overall  impact  of  an  ESS  shall  be  thoroughly  examined  before  implementation  to 
ensure  that  changes  do  not  result  in  additional  complexities,  loss  of  Situation  Awareness, 
or  possibilities  for  error. 

8.  The  ESS  shall  keep  an  up-to-date  record  of  where  the  operator  currently  is  within  a 
sequence  of  tasks  or  activities. 

9.  Organize  the  functionality  of  the  ESS  in  line  with  the  operator’s  tasks. 

10.  ESS-related  information  shall  be  structured  according  to  the  operator’s  task. 


Discussion: 

1.  An  operator-centered  design  process  is  a  highly  structured,  comprehensive  product 
development  methodology  driven  by  (1)  clearly  specified,  task-oriented  objectives  and  (2) 
recognition  of  operator  needs,  limitations  and  preferences.  Information  collected  using  this 
analysis  is  scientifically  applied  in  the  design,  testing  and  implementation  of  an  ESS. 
Defining  the  goals  and  functions  of  an  ESS  may  require  the  use  of  a  mission,  function  and 
task  analysis. 

2.  Input  from  the  operator  is  essential  in  defining  information  requirements.  To  increase  the 
likelihood  that  the  new  system  will  “fit”  most  operators  and  the  constraints  of  their  tasks,  a 
representative  number  of  operators  shall  be  involved  to  provide  advice  and  feedback  in  the 
development  of  the  system.  Not  only  shall  this  help  with  system  development,  but  it  shall 


2  These  guidelines  apply  to  the  whole  system  (and  not  just  the  ESS). 
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also  give  a  reasonable  number  of  operators  a  feeling  of  “ownership”  which  they  can 
transmit  to  their  colleagues,  thereby  helping  to  facilitate  the  development  of  trust. 

6.  Automation  of  tasks  may  deplete  team  interaction  and  cooperation  unless  all  parties  are 
provided  with  information  that  allows  them  to  be  actively  involved  in  the  task.  Task 
automation  can  cause  physical  difficulty  in  seeing  what  the  other  team  member  is  doing, 
reduce  the  ability  to  cross  monitor,  change  traditional  roles  and  responsibilities,  and 
change  the  manner  in  which  team  members  attempt  to  help  one  another. 

8.  This  allows  the  operator  to  resume  tasks  smoothly  and  efficiently  after  being  interrupted. 

9.  Information  objects  and  operations  shall  be  accessible  in  a  sequence  that  matches  the 
way  operators  will  most  effectively  and  productively  do  things  with  minimal  error. 


10.  Essential  information  that  is  regularly  needed,  cross-checked,  or  time-critical  should  be 
prominently  displayed.  Less  essential  information  can  be  less  prominently  displayed,  or 
minimized,  pending  examination  at  another  time. 


Evaluation  Measures  and  Methodoloqies: 

Design: 

Inspect 

Demonstrate 

Experiment 

•  Assess  design  methodology  is  compliant  with 
MIL-HDBK-46855A  (Human  Engineering 

Program  Process  and  Procedures). 

0 

□ 

□ 

•  Assess  overall  OMI  design  is  compliant  with 
COMDAT  OMI  Style  Guide  (and  other  relevant 
Human  Factors  standards)  and  usability 
‘heuristics’. 

0 

□ 

□ 

Operator  Acceptance: 

•  Subjective  assessment  of  the  adequacy  of 
system  usability  and  utility  by  Human  Factors 
professional  using  ‘walk-through’  or  heuristic 
analysis. 

□ 

□ 

0 

•  Questionnaire-based  assessments  of  the 
adequacy  of  operator’s  perceptions  of  system 
usability  and  utility 

□ 

□ 

0 

Relationship  to  Other  Guidelines: 


•  3. 3. 1.1  Employ  Operator-Centered  Principles 

•  3.3.2  Employ  Operator-centered  OMI  Design 
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3.5.2 


Adopt  Operator-Centered  Evaluation 
Principles _ 


Source:  HFDS,  2003;  DefStan,  1996. 


Guideline(s): 

1 .  Possible  interactions  with  other  tools,  system  functions,  and  operator  tasks  shall  be 
evaluated  when  a  new  ESS  is  designed. 

2.  New  ESS  components  shall  be  tested  with  the  complete  system,  including  other  system 
components  of  the  ESS,  to  ensure  they  function  together  as  an  effective  whole. 

3.  The  ESS  shall  be  tested  under  normal  modes  of  operation  and  under  failure  modes  of 
operation. 

4.  Contextually  valid  human-in-the-loop  experiments  and  simulations  shall  be  conducted  to 
validate  and  refine  the  ESS  design. 

5.  Evaluations  of  the  usability  of  the  system  shall  be  carried  out  at  all  phases  of  the  system 
development. 

6.  The  ESS  shall  be  tested  in  a  realistic  operational  environment  using  tasks  and  operators 
representative  of  the  final  system. 


Discussion: 

5.  These  evaluations  shall  be  used  both  to  assist  in  deciding  between  alternative  design 
options,  and  to  support  validation  that  the  design  satisfies  the  system’s  operability 
requirements. 

6.  The  tasks  shall  provide  reasonable  coverage  of  the  most  important  parts  of  the  system. 
The  test  tasks  can  be  designed  based  on  a  task  analysis  or  on  a  product  identity 
statement  listing  the  intended  uses  for  the  system. 


Evaluation  Measures  and  Methodoloqies: 

Inspect 

Demonstrate 

Experiment 

Design 

•  Assess  evaluation  methodology  is  compliant 
with  MIL-HDBK-46855A  (Human  Engineering 
Program  Process  and  Procedures) 

0 

□ 

□ 

Relationship  to  Other  Guidelines:  None 
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3.6  Training  and  Implementation 


3.6.1 


Manage  Introduction  of  the  ESS 


Source:  HFDS,  2003;  DefStan  1996; 
Parasuraman  and  Riley,  1997; 
Zachary  and  Ryder,  1997;  DISA, 
1996. 


Guideline(s): 

1.  An  ESS  shall  be  introduced  with  advanced  briefing  and  subsequent  training. 

2.  Before  the  ESS  is  introduced,  operators  shall  be  informed  of  associated  changes  and 
increases  in  the  work  effort,  as  well  as  the  benefits  associated  with  the  ESS. 

3.  Operators  shall  be  trained  to  acquire  an  adequate  understanding  (mental  model)  of  how 
the  ESS  works  in  order  to  use  it  effectively. 

4.  Training  programs  shall  stress  human-system  interaction  skills  and  cognitive/problem 
solving  skills  rather  than  psychomotor  skills. 

5.  When  the  ESS  requires  different  kinds  of  cognitive  processing,  ways  of  thinking,  and 
discarding  of  traditional  methods  and  skills,  then  training  shall  be  designed  to  address 
problems  related  to  these  changes. 

6.  Operators  shall  be  trained  on  what  constitutes  the  normal  ESS  output  so  that  the  operator 
can  easily  determine  whether  the  system  is  functioning  properly. 


Discussion: 

1 .  The  introduction  of  the  ESS  may  introduce  changes  in  traditional  roles  and  responsibilities, 
a  redistribution  of  authority  for  tasks  or  changes  to  the  nature  of  the  cognitive  demands 
imposed  on  the  human  operator. 

2.  The  roles  and  responsibilities  of  the  operators,  cognitive  demands,  and  operational 
procedures  may  change  as  a  result  of  introducing  automation. 

3.  Operators  must  possess  accurate  mental  models  of  the  system  in  order  to  use  it 
effectively,  comprehend  current  situations,  plan  their  actions,  predict  future  system  states 
and  diagnose  system  failures. 

4.  Problems  in  automation  may  not  be  inherent  in  the  technology  itself.  Problems  can  arise 
due  to  limitations  in  the  integration  of  the  operator  and  automation. 


Evaluation  Measures  and  Methodoloqies: 

Inspect 

Demonstrate 

Experiment 

Design: 

•  Training  plan  includes  Training  Needs  Analysis 

0 

□ 

□ 

(TNA). 

•  T raining  plan  compliant  with  MIL-HDBK-46855A 

0 

□ 

□ 
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guidance  for  training  (i.e.,  minimize  training 
requirements). 

•  Training  plan  includes  regular  training 
evaluations. 

0 

□ 

□ 

•  Subjective  assessment  of  adequacy  of 
embedded  training  system  (if  applicable). 

Operator  Acceptance: 

0 

□ 

□ 

•  Subjective  assessment  (i.e.,  questionnaire)  of 
the  adequacy  of  operator’s  acceptance  of  ESS. 

□ 

□ 

0 

•  Subjective  assessment  (i.e.,  questionnaire)  of 
the  adequacy  of  operator’s  perception  of  ESS 
usefulness. 

□ 

□ 

0 

Relationship  to  Other  Guidelines: 


•  3.6.2  Train  to  Overcome  ‘Automation  Biases’ 

•  3.6.3  Train  to  Overcome  ESS  Failures 

•  3.6.4  Promote  Operator  Acceptance  and  T rust  in  ESSs 


3.6.2 

Train  to  Overcome  ‘Automation  Biases’ 

Source:  HFDS,  2003:  DISA,  1996: 

Parasuraman  and  Riley,  1997. 

Guideline(s): 


1 .  Operators  shall  be  trained  to  recognize  inappropriate  uses  of  an  ESS  including  automation 
bias  (the  use  of  automation  in  a  heuristic  manner  as  opposed  to  actively  seeking  and 
processing  information). 

2.  Operators  shall  be  trained  to  recognize  and  understand  the  conditions  under  which  the 
system  may  be  unreliable,  and  to  learn  the  conditions  where  it  performs  well  (when  or 
when  not  to  question  the  automation). 

3.  Operators  shall  be  trained  not  to  become  overly  reliant  on  automation. 


Discussion: 

1.  There  are  different  categories  of  inappropriate  automation  use,  including  automation  bias, 
ignoring  or  turning  off  the  automation,  and  improper  implementation  of  automation. 
Operators  may  rely  on  decision  aids  in  a  heuristic  manner  (referred  to  as  automation  bias). 
Using  heuristics  is  to  apply  simple  decision-making  rules  to  make  inferences  or  to  draw 
conclusions  simply  and  quickly.  Heuristics  are  useful  principles  having  wide  application, 
but  may  not  be  strictly  accurate.  Usually  a  heuristic  strategy  is  optimal,  however,  under 
certain  conditions  heuristics  will  be  inappropriate  and  errors  or  misuse  may  occur. 
Automation  bias  leads  to  errors  of  omission  (failure  to  notice  system  anomalies  when 
automation  fails)  and  errors  of  commission  (acceptance  of  automated  decisions  without 
cross-checking  or  in  presence  of  contradictory  information).  Training  will  help  prevent 
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automation  bias  and  help  the  operator  learn  to  examine  multiple  sources  of  information 
before  making  a  decision.  Early  training  on  automation  bias  may  reduce  commission  errors 
for  operators  new  to  automation,  but  may  be  less  likely  to  reduce  omission  errors  or  errors 
made  by  expert  operators.  Inappropriate  use  of  automation  may  be  influenced  by  various 
individual  factors  such  as  self-confidence  in  completing  the  task,  trust  in  the  automation, 
differential  effects  of  fatigue,  and  how  all  of  these  factors  combined  weigh  into  the  decision 
making  process.  Inappropriate  use  of  automation  can  be  due  to  misuse  (automation  bias, 
complacency),  disuse  (ignoring  or  turning  off  automation)  or  abuse  (improper 
implementation  of  automation).The  roles  and  responsibilities  of  the  operators,  cognitive 
demands,  and  operational  procedures  may  change  as  a  result  of  introducing  automation. 

2.  Operators  must  learn  not  to  categorically  accept  the  recommendation  of  a  decision  aid. 
Understanding  the  automation’s  weaknesses  allows  operators  to  better  judge  how  much 
they  shall  trust  the  automation  without  becoming  overconfident  in  its  performance.  This 
recognition  process  may  impose  an  additional  workload  on  the  operator. 


3.  When  operators  rely  on  automation  too  much  they  become  susceptible  to  automation- 
induced  complacency.  Monitoring  failures  are  likely  to  occur  when  operators  become 
overly  reliant  on  automation. 


Evaluation  Measures  and  Methodoloaies: 

Design: 

Inspect 

Demonstrate 

Experiment 

•  Training  plan  includes  Training  Needs  Analysis 
(TNA) 

0 

□ 

□ 

Operator  Trust: 

•  Subjective  assessment  (i.e. ,  questionnaire)  of 
the  adequacy  of  operator  trust.  Excessive  levels 
of  trust  may  indicate  operator  complacency 
(over-trust)  and  very  low  levels  of  trust  may 
indicate  operator  scepticism. 

□ 

□ 

0 

Operator  Situation  Awareness: 

•  Subjective  assessment  (i.e.,  FASA 

questionnaire)  of  the  adequacy  of  the  quality  of 
the  operator’s  situation  assessment  processes 
(this  questionnaire  includes  items  relevant  to 
automation  bias). 

□ 

□ 

0 

Relationship  to  Other  Guidelines: 


•  3.6.1  Manage  Introduction  of  the  ESS 

•  3.6.3  Train  to  Overcome  ESS  Failures 

•  3.6.4  Promote  Operator  Acceptance  and  T rust  in  ESSs 
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3.6.3 


Train  to  Overcome  ESS  Failures 


Source:  HFDS,  2003;  AHCI,  1998; 
Endsley,  1996;  Parasuraman  and 
Riley,  1997. _ 


Guideline(s): 

1.  Operators  shall  be  trained  on  risk  assessment  and  actions  needed  for  risk  reduction. 

2.  Operators  shall  be  trained  on  transitioning  from  the  ESS  to  manual  operations. 

3.  Operators  shall  be  trained  to  ensure  proper  understanding  of  the  ESS’s  processes. 


Discussion: 

2.  If  the  ESS  was  to  fail,  operators  need  to  be  skilled  at  both  recognizing  the  failure  and 
taking  manual  control. 


Evaluation  Measures  and  Methodoloqies: 

Design: 

Inspect 

Demonstrate 

Experiment 

•  Training  plan  includes  Training  Needs  Analysis 
(TNA) 

0 

□ 

□ 

Operator  Performance: 

•  Objective  assessment  of  the  adequacy  of  the 
operator’s  ability  (e.g.,  speed  and  error)  to 
resume  manual  control. 

□ 

□ 

0 

•  Objective  assessment  of  the  adequacy  of  the 
operator’s  ability  (e.g.,  speed  and  error)  to 
detect  and  manage  ESS  failures. 

□ 

□ 

0 

Relationship  to  Other  Guidelines: 

•  3.6.1  Manage  Introduction  of  the  ESS 

•  3.6.2  Train  to  Overcome  ‘Automation  Biases’ 

•  3.6.4  Promote  Operator  Acceptance  and  T rust  in  ESSs 


i _ i 

3.6.4 

Promote  Operator  Acceptance  and  Trust  in 
ESSs 

Source:  HFDS,  2003:  DefStan,  1996: 
MS1472F;  Parasuraman  and  Riley, 
1997. 

Guideline(s): 

1 .  T raining  shall  be  provided  to  enable  the  operator  to  calibrate  their  trust  in  the  ESS. 

2.  Changes  in  cognitive  processing,  ways  of  thinking,  and  methods  and  skills  used  for  the 
ESS  shall  be  minimized. 
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3.  To  promote  operator  acceptance,  operators  shall  be  trained  to  ensure  proper 
understanding  of  the  system’s  processes. 


Discussion: 

1 .  T raining  will  allow  the  operator  to  develop  an  adequate  model  of  how  reliable  or  unreliable 
the  ESS  is  under  specific  conditions. 

2.  An  ESS  that  requires  different  kinds  of  cognitive  processing,  ways  of  thinking,  and 
discarding  of  traditional  methods  and  skills  may  cause  the  system  to  be  both  less  efficient 
and  less  acceptable  to  the  operators.  This  could  include  automatic  conversion  of  data  into 
a  usable  format. 


3.  The  better  the  operator  understands  the  ESS,  the  more  likely  the  operator  is  to  trust  the 
ESS  appropriately.  Designers  need  to  alter  the  false  belief  that  ESSs  are  perfect  and 
ensure  that  the  operators  understand  when  the  system  is  likely  to  become  unreliable. 


Evaluation  Measures  and  Methodoloqies: 

Design: 

Inspect 

Demonstrate 

Experiment 

•  Training  plan  includes  Training  Needs  Analysis 
(TNA) 

0 

□ 

□ 

•  Training  plan  includes  regular  training 
evaluations 

0 

□ 

□ 

Operator  Trust: 

•  Subjective  assessment  (i.e. ,  questionnaire)  of 
the  adequacy  of  operator  trust.  Excessive  levels 
of  trust  may  indicate  operator  complacency 
(over-trust)  and  very  low  levels  of  trust  may 
indicate  operator  scepticism. 

□ 

□ 

0 

Operator  Acceptance: 

•  Subjective  assessment  (i.e.,  questionnaire)  of 
the  adequacy  of  operator’s  acceptance  of  ESS. 

□ 

□ 

0 

•  Subjective  assessment  (i.e.,  questionnaire)  of 
the  adequacy  of  operator’s  perception  of  ESS 
usefulness. 

□ 

□ 

0 

Relationship  to  Other  Guidelines: 


•  3.6.1  Manage  Introduction  of  the  ESS 

•  3.6.2  Train  to  Overcome  ‘Automation  Biases’ 

•  3.6.3  T rain  to  Overcome  ESS  Failures 
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4  Conclusions 


4.1  Summary 

The  development  of  standards  and  guidelines  presented  in  this  document  are  intended 
specifically  to: 

•  Enhance  the  usability  of  the  ESS  technologies  developed  within  the  INCOMMANDS 
project;  and, 

•  To  minimize  the  probability  (and  impact)  of  operator  error  under  the  stressful 
conditions  of  the  battlefield  environment. 

In  general,  standards  and  guidelines  (which  include  the  present  document)  specify  the  Took’ 
and  ‘behaviour’  of  the  operator’s  interaction  with  the  decision  aid  OML  As  with  all 
guidelines,  those  that  are  optimal  for  one  situation  may  not  be  suitable  for  another  situation. 

As  a  consequence,  design  trade-offs  might  occur  that  contradict  one  particular  standard  in 
favour  of  supporting  another.  In  these  cases,  it  is  important  that  each  design  trade-off  is 
recorded  and  the  consequences  for  overall  system  performance  are  evaluated  in  consultation 
with  a  person  experienced  in  psychometric-based  evaluations. 

4.2  The  Way  Ahead 

The  present  document  represents  a  significant  step  towards  a  comprehensive  Eluman  Factors 
Design  and  Evaluation  Guide  to  support  and  inform  the  design  of  OMls  for  decision  aids,  and 
related  technologies,  used  within  the  context  of  naval  C2  applications.  In  addition,  the  guide 
will  include  general  guidance  regarding  how  to  evaluate  compliance  with  the  guidelines.  The 
following  sections  discuss  a  number  of  possible  ways  forward  for  the  OM1  design  guidelines 
and  the  evaluation  criteria. 

4.2.1  OMI  Design  Guidelines 

It  is  critical  that  developing  OMls  be  subject  to  continual  review  to  both  confirm  compliance 
(using  the  evaluation  criteria  outlined  in  this  document)  with  the  design  guidance  provided  in 
this  document,  and  identify  usability  problems  in  the  OML  As  successful  solutions  are 
developed,  they  should  be  captured  as  design  specifications  and  added  to  the  guidance 
provided  in  this  document.  It  is  therefore  imperative  that  the  document  is  iterated  in  line  with 
the  experimental  studies  within,  and  outside  of,  the  INCOMMANDS  project. 

4.2.2  Evaluation  Measures  and  Methodologies 

The  vast  majority  of  evaluation  measures  described  within  this  document  require  some  form 
of  human- in-the -loop  experimentation.  This  is  due  to  a  combination  of  the  imprecision  of  the 
constructs  that  need  to  be  measured  (e.g.,  Situation  Awareness,  workload  and  trust),  the 
imprecision  of  the  measurement  tools  used  to  measure  these  constructs,  and  the  imprecision  of 
the  guidelines  themselves.  For  example,  some  aspects  of  the  design  of  OMls,  such  as  font 
size,  colour  coding  and  menu  structures,  can  be  specified  in  detail  and  evaluated  using  solely 
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inspection  methods.  However,  most  of  the  guidelines  within  this  document  are  not  as  clearly 
specified  and  therefore  need  time-consuming  and  expensive  experimentation  methods  to 
determine  compliance.  It  is  therefore  imperative  that  any  lessons  learnt  from  one  OM1 
development  project  should  be  shared  with  the  wider  community.  For  example,  following 
experimental  studies  to  evaluate  compliance  of  a  candidate  OM1  to  these  guidelines,  the 
results  of  these  studies  should  be  captured  in  the  form  of  more  detailed  design  specifications 
and  evaluation  methods. 

4.2.3  Integration  with  COMDAT  OMI  Style  Guide 

The  next  stage  for  this  work  is  to  integrate  the  guidance  within  this  document  with  the 
guidance  provide  in  the  existing  COMDAT  OMI  Style  Guide  to  form  a  final  document,  the 
INCOMMANDS  Human  Factors  Design  and  Evaluation  Guide.  This  comprehensive  guide 
will  cover  both  the  design  of  the  OMI  and  decision  aid  guidance,  used  within  the  context  of 
naval  C2  applications.  Some  preliminary  recommendations  are  suggested  below  for 
integrating  the  two  documents: 

•  Structure  of  the  INCOMMANDS  Human  Factors  Design  and  Evaluation  Guide.  The 
guidance  within  this  document  comprises  guidelines  relevant  to  aiding  and  supporting 
operator  decision  making  as  well  as  promoting  high  levels  of  operator  trust  and 
acceptance  of  the  system.  The  COMDAT  OMI  Style  Guide  on  the  other  hand  provides 
guidance  to  create  a  common  look  and  feel  and  of  basic  principles  of  OMI  usability. 
While  both  style  guides  provide  OMI  design  guidance  that  is  complimentary  to  each 
other,  the  focus  of  each  style  guide  is  different  and  requires  that  these  guidelines  are 
treated  individually.  It  is  recommended  that  the  INCOMMANDS  Human  Factors 
Design  and  Evaluation  Guide  be  comprised  of  two  main  sections;  one  section  should 
address  guidelines  specific  to  supporting  decision  making  while  the  other  section  should 
provides  concrete  OMI  design  guidance. 

•  Ensure  consistency  among  the  guidelines  provided  in  this  document  with  those  provided 
in  the  COMDAT  Style  Guide.  Suggestions  for  additional  OMI  guidance  and/or  removal 
or  modifications  of  existing  guidance  in  this  document  and  from  the  COMDAT  Style 
Guide  shall  be  provided  to  ensure  that  all  guidelines  are  consistent. 

•  Cross-referencing  of  guidelines.  Where  appropriate,  the  guidelines  from  the  COMDAT 
Style  Guide  should  refer  to  the  guidelines  found  within  this  document  for  supporting 
justification  of  OMI  design  guidance  related  to  decision  aiding.  Similarly,  the  guidelines 
from  this  document  should  refer  to  concrete  OMI  design  guidelines  in  the  COMDAT 
Style  Guide  for  guidance  on  implementing  decision  aiding  principles. 

•  Structure  the  format  of  the  COMDAT  Style  Guide  to  emulate  the  guidelines  format  within 
this  document.  The  guidelines  within  COMDAT  Style  Guide  should  be  structured  to 
impose  a  consistent  and  logical  format  consistent  with  the  format  of  the  guidelines  in  this 
report.  Each  and  every  COMDAT  guideline  would  be  composed  of  a  summary  section 
followed  by  a  list  of  guidelines  relevant  to  the  topic.  To  ensure  compliance  with  the 
COMDAT  Style  Guide,  the  evaluation  measures  outlined  in  the  present  document,  in 
terms  of  metrics  and  tools,  would  be  used  for  the  evaluation  of  a  proposed  OMI. 
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6  Acronyms 


AWW 

Above  Water  Warfare 

BMC4I 

Battle  Management  Command,  Control,  Communications, 

C2 

Computers  and  Intelligence 

Command  and  Control 

CDSC 

Command  Decision  Support  Concept 

CFMWC 

Canadian  Forces  Maritime  Warfare  Centre 

COMDAT 

Command  Decision  Aiding  Technology 

CONOPS 

Concept  of  Operations 

CPF 

Canadian  Patrol  Frigate 

DEP 

Demonstration  and  Experimental  Plan 

DISA 

Defense  Infonnation  Systems  Agency 

DND 

Department  of  National  Defence 

DoD 

Department  of  Defence 

DSS 

Decision  Support  System 

ESS 

Electronic  Support  System 

FAA 

Federal  Aviation  Administration 

FASA 

Factors  Affecting  Situation  Awareness 

HFDG 

Human  Factors  Design  Guide 

HFDS 

Human  Factors  Design  Standard 

HFEPP 

Human  Factors  Engineering  Program  Plan 

HMCCS 

Halifax  Modernized  Command  and  Control  System 

INCOMMANDS 

Innovative  Naval  COMbat  MANagement  Decision  Support 

KBS 

Knowledge-Based  System 

LoL 

Levels  of  Automation 

MFTA 

Mission,  Function  and  Task  Analysis 

NASA-TLX 

National  Aeronautics  and  Space  Administration-Task  Load 

Index 

OODA 

Observe  Orient  Decide  Act 

OMI 

Operator  Machine  Interface 

ORO 

Operations  Room  Officer 

PEOU 

Perceived  Ease-of-Use 

PU 

Perceived  Usefulness 

SA 

Situation  Awareness 

SAGAT 

Situation  Awareness  Global  Assessment  Technique 

SART 

Situation  Awareness  Rating  Technique 

SWC 

Sensor  Weapons  Controller 

TADMUS 

Tactical  Decision  Making  Under  Stress 

TAM 

Technology  Acceptance  Model 

TDP 

Technology  Demonstration  Program 

TEWA 

Threat  Evaluation  and  Weapons  Assignment 

UCD 

User-Centered  Design 
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Annex  A:  Basic  OMI  Design  Requirements 


The  following  section  describes  general  usability  heuristics,  or  ‘rules  of  thumb’,  that  have 
been  developed  and  are  widely  accepted  within  the  human  factors  industry  (e.g.,  Nielsen, 
1994).  The  following  heuristics  are  considered  to  be  basic  requirements  for  complex  operator 
interfaces  and,  as  such,  can  be  used  as  over-arching  guidelines  throughout  the  development  of 
the  1NCOMMANDS  Human  Factors  Design  and  Evaluation  Guide: 

•  Visibility  of  system  status.  The  system  shall  always  keep  operators  informed  about  what  is 
going  on,  through  appropriate  feedback,  within  reason; 

•  Match  between  system  and  the  real  world.  The  system  shall  speak  the  operators’ 
language,  with  words,  phrases  and  concepts  familiar  to  the  operator,  rather  than  system- 
oriented  terms.  Real-world  conventions  shall  be  followed  to  make  information  appear  in  a 
natural  and  logical  order; 

•  Operator  control  and  freedom.  Operators  often  choose  system  functions  by  mistake  and 
will  need  a  clearly  marked  “emergency  exit”  to  leave  the  unwanted  state  without  having 
to  go  through  an  extended  dialogue.  Undo  and  redo  functionality  shall  be  supported  where 
practical; 

•  Error  prevention.  Careful  design  shall  prevent  problems  from  occurring  in  the  first  place; 
if  errors  do  occur,  appropriate  alarms  and  alerts  shall  be  presented  to  the  operator; 

•  Recognition  rather  than  recall.  Objects,  actions,  and  options  shall  be  visible  to  the 
operator  at  all  times.  The  operator  shall  not  have  to  remember  information  from  one  part 
of  the  dialogue  to  another.  Instructions  for  use  of  the  system  shall  be  visible  or  easily 
retrievable  whenever  appropriate; 

•  Flexibility  and  efficiency  of  use.  Accelerators  (e.g.,  hot-keys),  which  are  unseen  by  the 
novice  operator,  shall  be  used  to  speed  up  the  interaction  for  the  expert  operator.  In  this 
way,  the  system  can  cater  to  both  inexperienced  and  experienced  operators.  Operators 
shall  also  be  allowed  to  tailor  how  they  perform  frequent  tasks  (e.g.,  re -configure 
windows); 

•  Help  operators  recognize,  diagnose,  and  recover  from  errors.  Error  messages  shall  be 
expressed  in  plain  language,  precisely  indicate  the  problem,  and  constructively  suggest  a 
solution; 

•  Help  and  documentation.  Even  though  it  is  better  if  the  system  can  be  used  without 
documentation,  it  may  be  necessary  to  provide  help  and  documentation.  Any  such 
information  shall  be  easy  to  search,  focused  on  the  operator’s  task,  list  concrete  steps  to 
be  carried  out,  and  not  be  too  large;  and, 

•  Consistency  and  standards.  Operators  shall  not  have  to  wonder  whether  different  words, 
situations,  or  actions  mean  the  same  thing.  Platform  conventions  shall  be  followed  where 
practical.  C2  relies  on  data  being  presented  in  a  manner  that  contributes  to  the  operators’ 
knowledge  of  the  tactical  situation;  the  ability  to  access  information  and  to  convert  it  to 
knowledge  is  enhanced  if  the  OMI  is  consistent.  Consistency  in  the  OMI  design  permits 
the  operators  to  devote  their  attention  to  the  information  supplied  by  the  system  rather 
than  to  the  system  itself.  If  the  OMI  is  consistent  throughout  the  application,  and  is 
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consistent  with  other  computer  experiences,  then  the  operators  do  not  need  to  devote 
attention  to  details  of  operating  the  software  controls  and  displays.  Instead,  cognitive 
processing  is  focused  on  the  tactical  information. 

One  of  the  major  goals  of  the  INCOMMANDS  Human  Factors  Design  and  Evaluation 
Guide  is  to  enforce  consistency.  To  achieve  this  goal,  there  must  be  consistency  within  the 
OM1  and  between  OMls.  In  addition,  consistency  is  not  just  a  question  of  interface  design 
but  includes  the  consideration  of  tasks  and  functionality  structure  of  the  system  that  match 
the  tasks  and  decision  making  processes  of  the  operator.  Each  instance  of  inconsistency  is 
likely  to  produce  unnecessary  cognitive  processing  and  affect  the  cognitive  processing 
available  to  the  decision  maker  for  the  actual  combat  tasks.  The  1NCOMMANDS  OM1 
therefore  shall  be  designed  to  be  consistent;  appearing,  behaving  and  responding  the  same 
throughout.  The  following  is  a  list  of  the  different  types  of  consistency  that  shall  be 
considered: 

o  Consistency  with  the  current  Canadian  Patrol  Frigate  (CPF)  CCS  system. 

o  Consistency  with  existing  military  OMI  style  guides.  In  particular,  the  COMDAT  V2, 
AHC1  (1998),  D1SA  (1996),  and  MIL-STD  1472F. 

o  Visual  consistency  (general  OMI  layout  “look  and  feel”).  The  same  information  shall 
be  presented  in  the  same  location  on  all  screens  and  dialogue  boxes  and  it  shall  be 
formatted  in  the  same  way  to  facilitate  recognition.  Menus  shall  be  presented  in  a 
consistent  format  throughout  the  system  and  shall  be  readily  available  at  all  times. 

o  Consistency  with  operator  expectations.  Display-control  relationships  must  be 
compatible  with  the  operator’s  expectations,  and  require  minimum  processing  to 
extrapolate  the  information  from  the  system. 

o  Consistency  with  the  decision  maker’s  mental  representation.  The  problem 
representation  within  the  decision  aid  shall  reflect  the  problem  representation, 
cognitive  strategies  and  expectations  and  work  practices  of  the  decision  maker. 

o  Consistent  language/terminology.  Small  changes  in  the  language  lead  to  errors  and 
confusion.  Operators  assume  that  different  terms  reflect  differences  in  the  software. 
For  example,  the  term  “Close”  is  expected  to  result  in  a  different  action  than  “Exit”  so 
these  shall  not  be  used  to  label  the  same  action.  Conversely,  if  more  than  one  term  is 
used  to  convey  the  same  concept,  then  the  operator  must  determine  if  two  different 
terms  reflect  the  same  software  activity  or  a  different  software  activity.  For  example, 
an  application  may  incorrectly  use  three  notations:  “Stop”,  “Cease”,  and  “End”  each 
to  mean  that  the  processing  will  not  be  continued  (e.g.,  identical  terminology  shall  be 
used  in  prompts,  menus,  and  help  screens). 

o  Consistent  symbols  and  icons.  Using  more  than  one  icon  design  to  represent  instances 
of  a  single  type  of  control  will  lead  to  errors  and  confusion.  For  example,  using  a  door 
icon  and  an  “X”  symbol  both  to  indicate  “Close”  in  an  application  will  lead  the 
operator  to  assume  that  the  “X”  and  “Close”  operate  differently. 

o  Feedback  consistency.  The  interface  shall  have  a  reliable  and  consistent  method  of 
system  response  across  applications.  Transactions  made  by  the  operator  shall  produce 
a  consistent  perceptual  response  whether  it  is  in  visual,  tactile,  or  auditory  form. 
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